Monocular emotion recognition sparse Transform modeling method based on event driving

Abstract

The invention belongs to the technical field of computer vision, and provides a single-eye emotion recognition sparse Transform modeling method based on event driving. The invention provides an event camera-oriented efficient visual Transform framework for eye emotion recognition, and a window selection and double-mask token selection combined pruning optimization mechanism is innovatively designed for event data characteristics and periorbital region analysis requirements. The method comprises the following steps: firstly, dynamically screening out a low-value region by quantitatively evaluating the information density of a local window so as to reduce calculation redundancy, and strengthening attention focusing on periocular key features; on this basis, a decision model fusing event stream time-space characteristics and token semantic relevance is further constructed, precise recognition and dynamic pruning of redundant tokens are realized by using binary masks, the calculation complexity is remarkably reduced on the premise of ensuring the classification precision, and an efficient solution is provided for real-time emotion recognition scenes with limited resources. The method provided by the invention can have remarkable advantages in a monocular emotion recognition task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a sparse Transformer modeling method for single-eye emotion recognition based on event driving. BACKGROUND

[0002] Compared with face-based or electroencephalogram (EEG)-based methods, eye-based emotion recognition has significant advantages, such as non-invasiveness, enhanced privacy protection, and higher flexibility. Therefore, eye-based emotion recognition methods have broad application prospects in the fields of clinical diagnosis, driver fatigue assessment, and immersive virtual reality / augmented reality interaction. However, existing eye-based emotion recognition methods based on traditional RGB cameras still face challenges such as motion blur artifacts and limited dynamic range.

[0003] Unlike traditional cameras, event cameras have extremely high temporal resolution and dynamic range, which can effectively capture the rapid changes of eye movements and blinking, especially under low light conditions. However, event data has inherent asynchrony and sparsity, which makes the eye-based emotion recognition task based on event data more complex, and traditional convolutional neural networks (CNN) based on RGB images are difficult to handle such data.

[0004] In recent years, the Transformer architecture has shown good results in event-driven object classification and recognition tasks, but the computational complexity and memory requirements of its self-attention mechanism limit its application on resource-constrained devices. In addition, existing Transformer models have the problem of computational redundancy in the eye-based emotion recognition task, mainly due to two reasons: (i) spatial concentration: the eye and eyebrow regions are mainly located in the center of the image, and the peripheral regions contribute less to the emotional semantics; (ii) event sparsity: event cameras only respond to brightness changes, resulting in no event information in static regions.

[0005] Therefore, it is necessary to provide a sparse Transformer framework based on event data and eye-based emotion recognition tasks. SUMMARY

[0006] According to the above-mentioned technical problems, a sparse Transformer modeling method for single-eye emotion recognition based on event driving is provided. The present application constructs a sparse Transformer framework based on event data and eye-based emotion recognition tasks, combining two sparsification strategies: window selection (WS) and dual-mask token selection (DTS), thereby significantly improving the computational efficiency while ensuring the accuracy of emotion recognition.

[0007] The technical means adopted by the present application are as follows: An event-driven monocular emotion recognition sparse Transformer modeling method, comprising: Converting the original event stream data into continuous event frame images; Inputting a group of continuous event frame images into a Swin Transformer model architecture containing multiple stages for processing; Performing a window selection strategy in the first stage of the Swin Transformer model; Performing a double-masked token selection strategy in the third stage of the Swin Transformer model; Based on the output of the Swin Transformer model, predicting the emotion category score of the monocular region.

[0008] Further, the window selection strategy comprises: evaluating the importance of all local windows in the first stage, selecting the top % of the window with the highest importance score as the active window, performing subsequent multi-head self-attention and multi-layer perception calculation on the active window, and skipping multi-head self-attention and multi-layer perception calculation of the non-active window.

[0009] Further, when evaluating the importance of all local windows in the first stage, the norm of each local window feature representation is calculated, and the top % of the window with the highest norm score is selected as the active window.

[0010] Further, the non-active window skips multi-head self-attention and multi-layer perception calculation, and the processed features of the active window are recombined with the unprocessed non-active window through scattering operation, maintaining the spatial structure compatibility of the feature map.

[0011] Further, the double-masked token selection strategy specifically comprises: Generating a double-masked , which combines the structural prior information based on the input event data and the semantic context information based on the current layer token representation; using the double-masked to filter active tokens and prune non-active tokens; performing sparse multi-head self-attention calculation so that attention calculation is mainly or only performed on the active tokens.

[0012] Further, generating the double-masked specifically comprises: ​Based on the down-sampling representation of the original event stream data, a token sparsity score reflecting the global spatial structure prior is calculated, and an important token event mask is selected; the event mask is generated through the event mask branch

[0013] Based on the current layer input token, features are extracted through a split-transform-fuse module, and the extracted features are Gumbel-Softmax sampled to generate a predictor mask reflecting the importance of semantic context; the predictor mask is generated through the predictor mask branch ; The event mask and the predictor mask are fused into a double mask through Hadamard product operation .

[0014] Further, the event mask is generated, specifically including: The input event image is processed and down-sampled to align the feature resolution, and the L2-norm is used to calculate the event sparsity score The top K token indexes with the highest scores are selected, and the event mask is set to 1 at these index positions and 0 at the remaining positions.

[0015] Further, the predictor mask is generated, specifically including: The current layer token is applied to the structure processing containing channel splitting, transformation, global information extraction and fusion, and the prediction score is generated through Softmax , and the binary predictor mask is generated through Gumbel-Softmax sampling .

[0016] Further, the calculation of the sparse multi-head self-attention includes: According to the double mask , the attention score corresponding to the non-activated token to be pruned is applied to the punishment and then normalized through Softmax, so that the contribution of the non-activated token is shielded when the sparse multi-head self-attention output is calculated.

[0017] Compared with the prior art, the present application has the following advantages: ​The event-driven monocular emotion recognition sparse Transformer modeling method provided by the application combines window selection and double-mask token selection two sparse strategies, can efficiently extract emotion features of the eye region, and can still maintain excellent performance in low light and rapid eye movement complex scenes. Through the introduction of the sparse strategy, the calculation redundancy is significantly reduced, the use of memory and computing resources is optimized, and the computing efficiency is improved. At the same time, the method focuses on the key information area, maintains high emotion classification accuracy, and is especially suitable for eye emotion recognition applications that need real-time processing and deployment on resource-limited devices.

[0018] The event-driven monocular emotion recognition sparse Transformer modeling method provided by the application can effectively extract the spatio-temporal features in the eye emotion recognition task by using the high time resolution and dynamic range of the event camera, provide an efficient and accurate emotion recognition scheme, and is especially suitable for application scenarios such as clinical diagnosis, driver fatigue assessment, virtual reality / augmented reality interaction, etc.

[0019] For the above reasons, the application can be widely promoted in the field of computer vision technology. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 The event-driven monocular emotion recognition sparse Transformer modeling method flowchart in the application.

[0022] Figure 2 The double-mask generation process in the application. DETAILED DESCRIPTION

[0023] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0024] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The following description of at least one example embodiment is merely illustrative in nature and not intended to limit the present application and its applications or uses in any way. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0025] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a reference to the presence of a feature, step, operation, device, component and / or combinations thereof.

[0026] Unless specifically stated otherwise, the relative arrangements of components and steps, numerical expressions, and numerical values set forth in the various examples herein are not limiting. It should be clearly understood that, although the use of terms such as "comprise", "include", "have", or "contain" can refer to the presence of a feature, step, operation, device, component, or combination thereof, these terms are not intended to be construed as a limitation on the various embodiments described herein. It should also be clearly understood that, for ease of description and understanding, the dimensions of the various portions shown in the drawings are not drawn to scale. Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered as part of the specification where appropriate. In all examples shown and discussed herein, any specific value should be interpreted as merely illustrative and not as a limitation. Therefore, other examples of the exemplary embodiments can have different values. It should be noted that similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0027] As shown in Figure 1 The present application provides an event-driven monocular emotion recognition sparse Transformer modeling method. The core of the method is the sparse strategy introduced in different stages of the model. The design of these strategies is closely combined with the inherent characteristics of event data (spatio-temporal sparsity, a large number of background noise or invalid events) and the requirements of the eye emotion recognition task (focusing on the subtle changes around the eyebrows and eyelids, and considering real-time performance). By focusing on high-density event areas, suppressing noise interference, and selectively processing information, the method aims to improve efficiency and accuracy. The present application contains two key sparse strategies: window selection (WS) and dual-mask token selection (DTS).

[0028] The method of the present application comprises: converting raw event stream data into continuous event frame images; inputting a set of continuous event frame images into a Swin Transformer model architecture containing multiple stages for processing.

[0029] A window selection strategy is performed in the first stage of the Swin Transformer model; the hierarchical architecture of the Swin Transformer usually uses different numbers of windows in stages 1 to 4, for example, stage 1 uses 64 small windows, and the number of windows decreases and the size increases in subsequent stages. The small dense windows in stage 1 are most likely to cover background information unrelated to the core area of the eye. Therefore, only in stage 1 does the window selection strategy reduce computational redundancy effectively at an early stage.

[0030] In implementation, as a preferred embodiment of the present application, the importance of all local windows in the first stage is evaluated, and the top windows in the importance score are selected as active windows, and the multi-head self-attention and multi-layer perception calculations are performed on the active windows, and the multi-head self-attention and multi-layer perception calculations of the non-active windows are skipped.

[0031] When evaluating the importance of all local windows in the first stage, for all local windows in the first layer of the Swin Transformer first stage, the information density is calculated, the norm of each local window feature representation is calculated, and the top windows with the highest norm score are selected as active windows, and the active window set . . is a preset percentage hyperparameter that controls the strength of window pruning. Non-active windows skip multi-head self-attention and multi-layer perception calculations, and the processed features of active windows are recombined with unprocessed non-active windows through a scattering operation to maintain the spatial structure compatibility of the feature map.

[0032] In implementation, multi-head self-attention (MSA) and multi-layer perceptron (MLP) are only performed on the selected active windows . The specific calculation process is as follows:

[0033]

[0034] ​​Wherein, MSA (Multi-Head Self-Attention) refers to a (regular or shifted) multi-head self-attention module, MLP (Multilayer Perceptron) refers to a multi-layer perceptron, and LN (Layer Normalization) refers to layer normalization. is the feature of the activated window after multi-head attention processing, is the feature after multi-layer perceptron processing. The non-activated window with a lower norm score completely skips the calculation of the above MSA (Multi-Head Self-Attention) and MLP (Multilayer Perceptron).

[0035] In order to ensure that the data stream can be smoothly transmitted to the downstream layer (such as the next layer or the next stage) and keep the spatial structure integrity of the feature map, it is necessary to combine the activated window features calculated with the non-activated windows which do not participate in the calculation through a scatter operation according to their spatial positions in the original feature map.

[0036] Through the window selection mechanism, the number of windows participating in the complex self-attention calculation in the first stage is effectively reduced, and the model is guided to focus the computing resources on the high-information-value area (expected to be the eye movement area), so as to reduce the energy consumption while possibly improving the recognition accuracy.

[0037] The dual-mask token selection strategy is executed in the third stage of the Swin Transformer model; it aims to achieve more fine-grained sparsification and directly prunes the tokens (Token). DTS contains two key components: a dual-mask generator and a sparse multi-head self-attention. The purpose is to filter out tokens with no information or redundancy, so that the model can concentrate computing resources on the structure and emotion-related areas that are crucial to emotion recognition, thereby improving computing efficiency and recognition accuracy.

[0038] In specific implementation, as a preferred embodiment of the present application, the dual mask is generated combines the structure prior information based on the input event data and the semantic context information based on the token representation of the current layer; the dual mask filters the activated tokens and prunes the non-activated tokens; and the sparse multi-head self-attention calculation is performed so that the attention calculation is mainly or only performed on the activated tokens.

[0039] The dual mask Specifically, this includes: calculating token sparsity scores reflecting the prior global spatial structure based on the downsampled representation of the original event stream data; selecting important tokens to generate event masks; and generating event masks through event mask branches. Based on the input token of the current layer, features are extracted through the split-transform-fusion module. The extracted features are then sampled using Gumbel-Softmax to generate a predictor mask that reflects the importance of semantic context. The predictor mask is then generated through the predictor mask branch. ; event mask With predictor mask The two masks are fused together using the Hadamard product operation. .

[0040] The Event Mask Branch utilizes input information to provide a global structural prior, helping to quickly locate potential regions of interest. The input is the initial event frame image. First, the input is processed by a multilayer perceptron (MLP), possibly combined with average pooling, to divide it into blocks and initially convert it into token representations; then, it undergoes two downsampling operations. (For example, convolution or pooling with a stride of 2), align the resolution represented by the token to the feature resolution of the current stage (stage 3) to obtain the event token. :

[0041] Next, adopt -norm (e.g.) or Norm) to calculate these event tokens Event sparsity score Used to measure the intensity of event activity at each token location; generates a [database name] with [other data]. All-zero mask with consistent spatial dimensions Find the score The highest Index of tokens :

[0042] mask The middle corresponds to The position is set to 1:

[0043] Received This is the final event mask generated based on the input event data.

[0044] Generate event mask Specifically, this includes: processing and downsampling the input event image to align with the feature resolution, using... - Norm calculation of event sparsity score Select the highest scorer Each token index, and the event mask. Set the index to 1 at these index positions and 0 at the rest.

[0045] The predictor mask branch is based on the input tokens of the current layer (stage 3). This is used to generate a mask, aiming to adaptively determine the importance of tokens based on the semantic context learned by the network, thereby reducing the risk of suboptimal decisions that may result from relying solely on prior input. The tokens from the current layer are then used as input. A "split-transform-fusion" structure is used to process... First, it is processed by a multilayer perceptron (MLP), and then channel splitting is performed. The feature channel is divided into two parts. :

[0046] in, It is a feature that contains local information. It is a feature that contains global information. (Regarding...) Application channel average pooling operation To obtain global statistical information; to preserve detailed features. With global information extraction Stitch along the channel dimension (using (This is represented as a multilayer perceptron, which is then fused with another multilayer perceptron (MLP) to obtain the fused features.) :

[0047] Fusion features Apply the Softmax function to the token dimension to generate a predicted importance score for each token. To obtain a differentiable binary mask, the Gumbel-Softmax technique is employed, based on the predicted score. Perform sampling to generate a predictor mask. : .

[0048] Generate predictor mask Specifically, this includes: tokens at the current layer. The application includes structural processing such as channel splitting, transformation, global information extraction, and fusion, and generates prediction scores through Softmax. Then, Gumbel-Softmax sampling is used to generate a binary predictor mask. .

[0049] Final double mask By event mask and predictor mask Through element-wise Hadamard product, )get:

[0050] exist In this model, a value of 1 indicates that the token is retained, and a value of 0 indicates that the token is pruned. This fusion mechanism integrates prior structural information of the input data with the network's own semantic awareness capabilities, achieving more precise and adaptive token pruning.

[0051] In a specific implementation, as a preferred embodiment of the present invention, the calculation of sparse multi-head self-attention includes: based on the double mask... The attention scores corresponding to the pruned inactive tokens are penalized and then normalized using Softmax, thereby masking the contribution of inactive tokens when computing sparse multi-head self-attention output.

[0052] Standard multi-head self-attention (MSA) requires computing the query vector over all tokens (even those masked to zero). Query), key vector ( Key), value vector ( This not only wastes computational resources, but invalid tokens set to zero can still affect the representation learning of active tokens through the attention mechanism.

[0053] To address this, the present invention proposes sparse multi-head self-attention, the core of which is to only focus on the double masked signal. Perform valid attention computation on the tokens marked as reserved. One specific implementation is as follows: After calculating the attention weights and before applying the Softmax function, the attention score matrix is... Add a large negative mask The negative mask By double mask generate:

[0054] in, is a sufficiently large positive value. The mask needs to match the shape of the attention score matrix. In positions where the value is 0 (pruned), the value of is ; in positions where the value is 1 (kept), the value of is 0; the modified attention is computed as follows:

[0055] where is the final normalized sparse attention weights.

[0056] In this way, the attention scores corresponding to pruned tokens are added a very large negative value before Softmax, resulting in them being almost zero in the normalized sparse attention weights . This effectively masks the attention contribution of invalid tokens and prevents them from receiving information from other tokens. At the same time, this method maintains the fixed number of tokens within a batch, avoiding the implementation complexity brought by dynamic size adjustment, and avoiding unintended interactions between kept and pruned tokens.

[0057] Based on the output of the Swin Transformer model, the score of the emotion category of the monocular region is predicted.

[0058] In order to effectively train the sparse model proposed in the present application and balance the classification accuracy, knowledge distillation, and adaptive control of the sparsity degree of the model, a multi-task loss framework is adopted. The total loss function is composed of the following parts: Main loss : standard cross-entropy loss, used to supervise the model to learn discriminative features to complete the n-class emotion classification task.

[0059] Distillation loss : using knowledge distillation technology, a pre-trained, fixed standard (non-sparse) Swin Transformer is used as a teacher model. This loss term aims to align the intermediate layer token predictions (e.g. output tokens of stage 3) of the sparse model (student model) of the present application with the token predictions of the corresponding layer of the teacher model. In this embodiment, distillation loss (mean square error) can be used to calculate the difference between the student and teacher token representations.

[0060] KL divergence loss ​: This loss term is used to minimize the KL divergence of the final outputs (classification score vectors) of the student and teacher models, prompting the prediction distribution of the student model to imitate that of the teacher model.

[0061] Token retention rate loss : This loss term is used to explicitly control the actual token retention rate generated by Dual-Mask Token Selection (DTS). It constrains the deviation between the actual retention rate of tokens (obtained by averaging the dual-mask values of all tokens in the batch) and the preset target retention rate in the stage of applying Dual-Mask Token Selection (DTS) through Mean Squared Error (MSE). The specific calculation formula is as follows:

[0062] wherein, is the batch size, is the number of stages of applying Dual-Mask Token Selection (DTS), is the total number of tokens in the stage, is the dual-mask value (0 or 1) of the token of the sample in the stage, is the target token retention rate (hyperparameter) of the stage.

[0063] The calculation formula of the total loss is as follows:

[0064] wherein, , , are the weight coefficients of each loss term, used to balance the importance between them. In this example, they are set to .

[0065] By optimizing this integrated loss function, the model can learn the classification task while maintaining competitiveness with the help of knowledge distillation, and dynamically optimize the computational efficiency (i.e., token retention rate) according to the constraints.

[0066] ​​Those skilled in the art should understand that the above embodiments are only exemplary, and various modifications and combinations can be made without departing from the spirit and scope of the present application. For example, window selection and DTS can be applied to other stages of the model, evaluation indicators (such as norms, norms) and specific network structures (such as the number of layers and channels of a multi-layer perceptron) can be adjusted, and the specific form and weight of the loss function can also be optimized according to actual applications. Therefore, the scope of protection of the present application should be defined by the appended claims.

[0067] Embodiments The present application proposes a single-eye emotion recognition sparse Transformer modeling method based on event-driven, and the overall architecture is based on Swin Transformer. The architecture includes four processing stages (stage 1 to stage 4), and a downsampling operation is performed before each stage to reduce the feature map resolution and increase the receptive field. The original event stream data is first processed and mapped to a grid-based frame representation suitable for inputting into a neural network. This representation is then input into the constructed Swin Transformer model. The final output of the model is a score of a predefined emotion class, which is used for emotion classification prediction.

[0068] Figure 1 The flow framework diagram of the method of the present application includes three processes: (a) the overall architecture, the event stream is mapped to a frame graph, and after image block embedding (Patch Embedding), it is input into the four-stage Swin Transformer and the emotion class is output by the classifier; (b) the first stage window selection, according to norms, high information windows are selected, and only active windows are executed for multi-head self-attention and multi-layer perceptron; (c) the third stage double-mask token selection, generating and fusing event masks and predictor masks, and applying sparse multi-head self-attention only to the reserved tokens.

[0069] Figure 2 The double-mask generator process designed by the present application. The event mask branch and the predictor mask branch generate two masks in parallel, and the double mask obtained after fusion is used to screen active tokens.

[0070] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An event-driven sparse Transformer modeling method for monocular emotion recognition, characterized in that, include: Convert the raw event stream data into a continuous image of event frames; A set of consecutive event frame images is input into a multi-stage Swing Transformer model architecture for processing; The window selection strategy is executed in the first stage of the Swin Transformer model. A double-mask token selection strategy is executed in the third stage of the Swin Transformer model; Based on the output of the Swin Transformer model, the emotion category score of a single eye region is predicted.

2. The event-driven sparse Transformer modeling method for monocular emotion recognition according to claim 1, characterized in that, The window selection strategy includes: assessing the importance of all local windows in the first phase and selecting those with the highest importance scores. The window with a percentage of the active window is used as the active window. Subsequent multi-head self-attention and multi-layer perception calculations are performed on the active window, while multi-head self-attention and multi-layer perception calculations are skipped for the inactive window.

3. The event-driven sparse Transformer modeling method for monocular emotion recognition according to claim 2, characterized in that, When evaluating the importance of all local windows in the first phase, the feature representation of each local window is calculated. norm, and select The top norm score The % window is used as the active window.

4. The event-driven sparse Transformer modeling method for monocular emotion recognition according to claim 2, characterized in that, The inactive window skips multi-head self-attention and multi-layer perception computation, and recombines the features processed by the active window with the unprocessed inactive window through scattering operations, maintaining the spatial structural compatibility of the feature map.

5. The event-driven monocular emotion recognition sparse Transformer modeling method according to claim 1, characterized in that, The dual-mask token selection strategy specifically includes: Generate double mask The double mask Combining structural prior information based on input event data and semantic context information based on the token representation of the current layer; using the dual mask. Filter active tokens and prune inactive tokens; perform sparse multi-head self-attention computation so that attention computation is performed primarily or only on the active tokens.

6. The event-driven sparse Transformer modeling method for monocular emotion recognition according to claim 5, characterized in that, Generate the double mask Specifically, it includes: Based on the downsampled representation of the original event stream data, a token sparsity score reflecting the prior global spatial structure is calculated, and important tokens are selected to generate an event mask; an event mask is then generated through event mask branching. ; Based on the current layer input token, features are extracted through a split-transform-fusion module. The extracted features are then subjected to Gumbel-Softmax sampling to generate a predictor mask that reflects the importance of semantic context. A predictor mask is then generated through a predictor mask branch. ; mask the event With the predictor mask The two masks are fused together using the Hadamard product operation. .

7. The event-driven sparse Transformer modeling method for monocular emotion recognition according to claim 6, characterized in that, Generate the event mask Specifically, it includes: The input event image is processed and downsampled to align with the feature resolution, using... - Norm calculation of event sparsity score Select the highest scorer Each token index, and the event mask. Set the index to 1 at these index positions and 0 at the rest.

8. The event-driven sparse Transformer modeling method for monocular emotion recognition according to claim 6, characterized in that, Generate the predictor mask Specifically, it includes: For the current layer token The application includes structural processing such as channel splitting, transformation, global information extraction, and fusion, and generates prediction scores through Softmax. Then, Gumbel-Softmax sampling is used to generate a binary predictor mask. .

9. The event-driven sparse Transformer modeling method for monocular emotion recognition according to claim 5, characterized in that, The calculation of the sparse multi-head self-attention includes: According to the double mask The attention scores corresponding to the pruned inactive tokens are penalized and then normalized using Softmax, thereby masking the contribution of inactive tokens when computing sparse multi-head self-attention output.

Images