Multi-dimensional cross-attention based 3D face video emotion anomaly recognition method and device

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a multi-dimensional cross-attention 3D face video emotion anomaly recognition method and device. The present application uses the attention framework to fuse the RGB visual features and the facial depth features by the depth fusion of the space-time channel three-dimensional features; the multi-head self-attention mechanism is used to capture the long-distance and long-time sequence dependency in three dimensions, the dynamic weighted fusion of different dimensional features is realized by the cross-gating mechanism, the discriminability of the feature expression is effectively enhanced, the redundant feature interference is inhibited while the information integrity is maintained. Therefore, the feature extraction of the facial expression muscle dynamic change is optimized, the cross-modal complementary representation of the RGB texture and the depth geometric features is realized, and the emotion anomaly recognition is more accurate.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and device for recognizing emotional anomalies in 3D facial videos based on multidimensional cross-attention. Background Technology

[0002] Identifying emotional abnormalities is helpful for screening and early intervention of mental illnesses. Traditional methods mainly rely on psychological scales and clinical interviews, which suffer from problems such as strong subjectivity, high concealment, and long diagnostic time. In recent years, artificial intelligence-based facial video analysis technology has provided a new approach for the objective identification of emotional abnormalities.

[0003] However, existing methods mainly rely on 2D-RGB video data, using only visual features, which is insufficient for three-dimensional dynamic perception and makes it difficult to model the three-dimensional dynamic characteristics of facial muscle movements. In terms of modeling methods, most methods separate and extract spatiotemporal features and perform modal fusion by feature concatenation and weighting, ignoring cross-modal complementarity and making it difficult to characterize micro-expression transients and facial geometric deformations related to emotional abnormalities.

[0004] Therefore, there is a need for a method and device for identifying emotional anomalies in 3D face videos based on multidimensional cross-attention that can deeply explore cross-modal complementarity and achieve higher recognition accuracy. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned shortcomings in the prior art and to provide a method and device for identifying emotional anomalies in 3D face videos based on multidimensional cross-attention.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] A method for emotion anomaly recognition in 3D face videos based on multidimensional cross-attention includes the following steps:

[0008] S1: Acquire the face data to be analyzed, perform preprocessing, and output feature data; the face data to be analyzed includes the face video to be analyzed and the corresponding depth information, and the feature data includes face RGB features and depth information features;

[0009] S2: Input the feature data into the pre-trained TSC-CGAM model;

[0010] S3: The TSC-CGAM model outputs the emotion anomaly recognition result of the feature data;

[0011] The TSC-CGAM model uses a convolutional neural network model as its backbone structure, and inserts a TSC-CGA unit after each feature extraction unit in the convolutional neural network model; the TSC-CGA unit includes a temporal attention module, a spatial attention module, a channel attention module, and a multidimensional cross-gated fusion module.

[0012] As a preferred embodiment of the present invention, the preprocessing includes face region cropping, face RGB and depth information alignment, feature normalization, and feature extraction processing.

[0013] As a preferred embodiment of the present invention, the time attention module includes the following operating flow:

[0014] The feature data is subjected to temporal feature reconstruction processing; a learnable one-dimensional temporal location code is added to the input tensor after feature reconstruction processing, and the face RGB features and depth information features with temporal location code are output.

[0015] The query is generated by performing three independent linear transformations on the RGB features and depth information features of the face with time location encoding. ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get Each head focuses on a different temporal pattern; then, based on the correlation between each head in the temporal dimension, the weights associated with each head are obtained, and weighted fusion is performed; the outputs of all heads are concatenated and then fused with temporal information through a fully connected layer; finally, the output is reconstructed to the original spatial dimension to obtain the temporal output features extracted by the temporal attention mechanism; the expression for calculating the temporal output features is as follows:

[0016] Calculate the association weights and perform weighted fusion:

[0017] ,

[0018] Calculation of time-related information features:

[0019] ,

[0020] Time output feature calculation:

[0021] ,

[0022] in, The attention feature output for the Nth head. () is the activation function. These represent the batch size, channel dimension, time series length, feature height, and width of the input features, respectively, where D is the number of channels for each head. Features after fusing time information, For feature splicing processing, , For learnable weight matrix, For bias vectors, For feature reconstruction operations, This is a time-based output feature.

[0023] As a preferred embodiment of the present invention, the spatial attention module includes the following operating flow:

[0024] The feature data is subjected to spatial feature reconstruction processing; the two-dimensional spatial location code is decomposed into horizontal and vertical direction codes, and a learnable spatial location code is generated by adding them through an outer product; the learnable spatial location code is added to the input tensor after feature reconstruction processing, and the output is a face RGB and depth information feature with spatial location code; the expression of the learnable spatial location code is:

[0025] ,

[0026] in, For learnable spatial location encoding, Encoding in the horizontal direction, Encoding in the vertical direction, They respectively indicate that it is a length of or A vector of all 1s Indicates Kronecker product

[0027] A query is generated from the spatially encoded RGB and depth features of a face through three independent linear transformations. ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get Each head focuses on different spatial patterns; then, the correlation weights in the spatial dimension of each head are independently calculated and weighted fusion is performed; the outputs of all heads are concatenated and then fused through a fully connected layer to integrate spatial information; finally, the output is reconstructed to the original spatial dimension to obtain the spatial output features extracted by the spatial attention mechanism; the expression for the calculation process of the spatial output features is as follows:

[0028] Calculate the association weights and perform weighted fusion:

[0029] ,

[0030] Fusion spatial information feature calculation:

[0031] ,

[0032] Spatial output feature calculation:

[0033] ,

[0034] in, Let S be the attention feature output of the Nth head, where S is the spatial feature dimension and D is the number of channels per head. , To integrate spatial information and its features, For feature splicing processing, , For learnable weight matrix, For bias vectors, This is the spatial output feature.

[0035] As a preferred embodiment of the present invention, the channel attention module includes the following operating flow:

[0036] The feature data is processed through orthogonally initialized 3D convolution kernels. Redundant channel information is filtered out and compressed along the spatiotemporal dimension to extract the channel statistical feature tensor, whose expression is:

[0037] ,

[0038] ,

[0039] in, convolution kernel The filtered input tensor, where X is the input tensor. To achieve a mapping from an input of arbitrary size to an output of fixed size by performing weighted summation and scaling transformation on the input features, Input feature tensors into the processed channels;

[0040] The channel statistical feature tensor generates a query through three independent linear transformations. ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get Each head focuses on different channel modes; then, attention weights are calculated independently for each head; the attention weights of all heads are concatenated and fused with channel information through a fully connected layer to obtain channel weights; finally, the channel weights are multiplied by the input tensor after generating a channel mask through an activation function to obtain the channel output features after feature importance allocation in the channel dimension; the expression for the calculation process of the channel output features is as follows:

[0041] Calculate the association weights and perform weighted fusion:

[0042] ,

[0043] Calculation of fused channel information features:

[0044] ,

[0045] Channel output characteristic calculation:

[0046] ,

[0047] in, Let C be the attention feature output of the Nth head, where C is the channel feature dimension and D is the number of channels per head. , The channel weights after integrating channel information, For feature splicing processing, , For learnable weight matrix, For bias vectors, For channel output characteristics, This is the Sigmoid activation function.

[0048] As a preferred embodiment of the present invention, the multi-dimensional cross-gating fusion module includes the following operating flow:

[0049] The RGB and depth features of the face extracted by the temporal, spatial, and channel attention modules are learned by learning the weights of each dimension through convolutional layers. The outputs of the three modules are then gated and weighted, and finally normalized through residual connections to obtain the fused features of the temporal, spatial, and channel dimensions. ;

[0050] The expression for the fusion feature of the three dimensions of time, space, and channel is:

[0051]

[0052] in, The result is the fusion of the outputs from the three dimensions, where θ represents the learnable parameters, and FusionFunc is a non-linear mapping function. For time-based output features, For spatial output features, This refers to the channel output characteristics.

[0053] As a preferred embodiment of the present invention, the TSC-CGAM model is trained using labeled face videos and depth information datasets, and after convergence, a TSC-CGAM model capable of recognizing emotional anomalies is obtained.

[0054] As a preferred embodiment of the present invention, S4 is also included:

[0055] S4: Extract one or more of the time output features, spatial output features, channel output features, and three-dimensional fusion features from the TSC-CGA unit, convert them into a visual heatmap, and overlay the visual heatmap onto the original face RGB image to achieve feature map visualization.

[0056] As a preferred embodiment of the present invention, the conversion of the visualized heatmap includes the following steps:

[0057] Calculate the gradient weights of the features to be transformed;

[0058] Calculate the 3D heatmap of the feature to be converted based on the gradient weights;

[0059] The 3D heatmap is compressed into a 2D heatmap and output as a visual heatmap.

[0060] A device for recognizing emotional anomalies in 3D facial videos based on multidimensional cross-attention includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform any of the aforementioned methods for recognizing emotional anomalies in 3D facial videos based on multidimensional cross-attention.

[0061] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0062] This invention utilizes a proposed attention framework to deeply fuse RGB visual features with facial depth features across three dimensions of spatiotemporal channels. Furthermore, it captures long-distance and long-temporal dependencies through a multi-head self-attention mechanism across these three dimensions, and achieves dynamic weighted fusion of features from different dimensions through a cross-gating mechanism. This effectively enhances the discriminative power of feature representation while maintaining information integrity and suppressing redundant feature interference. Consequently, it optimizes feature extraction of dynamic changes in facial expression muscles, achieving cross-modal complementary representation of RGB texture and depth geometric features, and significantly improving the accuracy of emotional anomaly recognition. Attached Figure Description

[0063] Figure 1This is a flowchart illustrating a method for identifying emotional anomalies in 3D face videos based on multidimensional cross-attention, as described in Embodiment 1 of the present invention.

[0064] Figure 2 This is a schematic diagram of the TSC-CGAM model in the 3D face video emotion anomaly recognition method based on multidimensional cross-attention described in Embodiment 2 of the present invention;

[0065] Figure 3 This is a schematic diagram of the TSC-CGA unit in the 3D face video emotion anomaly recognition method based on multidimensional cross-attention described in Embodiment 2 of the present invention;

[0066] Figure 4 This is a schematic diagram of the structure of a 3D face video emotion anomaly recognition device based on multidimensional cross-attention, which utilizes the multidimensional cross-attention 3D face video emotion anomaly recognition method described in the foregoing embodiments, as described in Embodiment 5 of the present invention. Detailed Implementation

[0067] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0068] Example 1

[0069] like Figure 1 As shown, a method for emotion anomaly recognition in 3D face videos based on multidimensional cross-attention includes the following steps:

[0070] S1: Acquire the face data to be analyzed, perform preprocessing, and output feature data; the face data to be analyzed includes the face video to be analyzed and the corresponding depth information, and the feature data includes face RGB features and depth information features.

[0071] S2: Input the feature data into the pre-trained TSC-CGAM model (Temporal-Spatial-ChannelCross-Gated Attention Model).

[0072] S3: The TSC-CGAM model outputs the emotion anomaly recognition result of the feature data.

[0073] The TSC-CGAM model uses a convolutional neural network model as its backbone structure, and inserts a TSC-CGA unit after each feature extraction unit in the convolutional neural network model. The TSC-CGA (Temporal-Spatial-Channel Cross-Gated Attention) unit includes a temporal attention module, a spatial attention module, a channel attention module, and a multi-dimensional cross-gated fusion module.

[0074] Example 2

[0075] This embodiment is a specific implementation of the multidimensional cross-attention-based 3D face video emotion anomaly recognition method described in Embodiment 1, including the following steps:

[0076] S1: Acquire the face data to be analyzed, perform preprocessing, and output feature data.

[0077] The preprocessing includes face region cropping, face RGB and depth information alignment, feature normalization, and feature extraction; the face data to be analyzed includes the face video to be analyzed and the corresponding depth information, and the feature data includes face RGB features and depth information features.

[0078] Furthermore, in this step, during the acquisition process, an RGB video stream and facial depth information are simultaneously captured by a 3D camera to obtain a three-dimensional facial video of the subject. The depth data can quantify three-dimensional dynamic features such as nasolabial fold deformation and zygomatic muscle contraction amplitude, making up for the shortcomings of traditional 2D data in geometric motion modeling.

[0079] In this embodiment, after acquiring the face video, facial landmark detection is first performed on the acquired face video. The detected face region is aligned with the depth information image and uniformly acquired to a preset size. To ensure that the data dimension of the face depth image is consistent with the data dimension of the face video image, normalization is performed, and the output is RGB-D data (i.e., face RGB image and depth information). The RGB-D data is then input into a pre-constructed convolutional neural network for feature extraction, outputting face RGB features and corresponding depth information features.

[0080] S2: Input the feature data into the pre-trained TSC-CGAM model.

[0081] The TSC-CGAM model captures the temporal dynamic changes and spatial distribution characteristics of facial micro-expressions through spatiotemporal attention, and fuses RGB visual features and facial depth features using channel attention. The spatiotemporal and channel 3D attention mechanisms employ a dynamic gating fusion strategy to uncover the nonlinear complementary relationship between video RGB and depth features.

[0082] Furthermore, in this embodiment, the convolutional neural network in the TSC-CGAM model can be a ResNet model, a VGG model, or a DenseNet model. For example, when the convolutional neural network is a ResNet model, the TSC-CGA units are inserted before and after each residual module. When the convolutional neural network is a VGG model, the TSC-CGA units are inserted before each max-pooling layer. When the convolutional neural network is a DenseNet model, the TSC-CGA units are inserted before each transformation layer.

[0083] Furthermore, such as Figure 2 As shown, in this embodiment, the TSC-CGAM model is a ResNet model. The model inputs feature data, and the TSC-CGA unit replaces the intermediate layer of the bottleneck structure, generating a weighted feature with deep fusion of multi-dimensional features. After multiple residual modules, an adaptive pooling layer resamples the features into a fixed shape. Finally, a fully connected layer predicts the sentiment anomaly score as the final output of TSC-CGAM.

[0084] Furthermore, the TSC-CGAM model is trained using labeled face videos and depth information datasets. After convergence, a TSC-CGAM model capable of recognizing emotional anomalies is obtained. The video dataset includes the face videos to be analyzed and the corresponding depth information; specifically, the model training includes the following steps:

[0085] a: To analyze facial videos and their corresponding depth information, and to perform preprocessing;

[0086] b: Input the preprocessed face video to be analyzed and the depth information into a pre-constructed convolutional neural network for feature extraction, and output the face RGB features and the corresponding depth information features;

[0087] c: Input the RGB features and depth information features of the face into the pre-constructed TSC-CGA multidimensional cross-gated attention mechanism unit to fuse the RGB features and depth information features of the face;

[0088] The TSC-CGA unit includes a temporal attention module, a spatial attention module, a channel attention module, and a multidimensional cross-gating fusion module. It extracts the temporal, spatial, and channel features corresponding to the RGB features and depth information features of the face, and inputs them into the multidimensional cross-gating module for feature fusion.

[0089] d: The fused features output by the TSC-CGA multidimensional cross-gated attention mechanism unit are fed into the fully connected layer to output the final emotion anomaly recognition result.

[0090] Furthermore, it also includes training the TSC-CGAM model based on the degree of emotional abnormality of different users and adjusting the parameters of the TSC-CGAM model.

[0091] e: Obtain the trained TSC-CGAM model; extract the temporal features, spatial features, channel features, and multidimensional cross-gated attention mechanism units of the TSC-CGAM model, and visualize the features after multidimensional cross-gated fusion.

[0092] In the TSC-CGA unit, the spatial attention module, temporal attention module, and channel attention module generate internal features and attention vectors, such as Figure 3 As shown.

[0093] 1. Time-related module:

[0094] Temporal attention employs a multi-head attention mechanism, combining the advantages of the Transformer architecture to construct a long temporal dependency modeling framework, automatically focusing on key frames with drastic facial expression changes. The specific implementation steps are as follows: Step 1, Input feature reconstruction and position encoding:

[0095] The feature data is subjected to temporal feature reconstruction processing; a learnable one-dimensional temporal location code is added to the input tensor after feature reconstruction processing, and the face RGB features and depth information features with temporal location code are output.

[0096] Furthermore, this embodiment will input tensors Spatial dimension Merging into the batch dimension makes the features become This facilitates parallel computation on time series data; These represent the batch size, channel dimension, time series length, feature height, and width of the input features, respectively; the input dimensions are transformed during feature reconstruction processing. Add learnable one-dimensional temporal location coding The location of each time frame is recorded and aligned with the dimension through a broadcast mechanism to enhance temporal location awareness, resulting in processed temporal input features. Its expression is:

[0097] ,

[0098] The second step is to calculate the time-based multi-head self-attention mechanism:

[0099] The query is generated by performing three independent linear transformations on the RGB features and depth information features of the face with time location encoding. ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get Each head focuses on a different time pattern; then, based on the correlation between each head in the time dimension, the weight associated with each head is obtained, and weighted fusion is performed; the outputs of all heads are concatenated and then fused with time information through a fully connected layer; finally, the output is reconstructed to the original spatial dimension to obtain the time output features extracted by the time attention mechanism.

[0100] Specifically, each type of feature includes the following processing steps:

[0101] Time input features Generate a query through three independent linear transformations ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get To focus on different time patterns; then, independently calculate the correlation weights on the time dimension for each head and perform weighted fusion;

[0102] ,

[0103] in, The attention feature output for the Nth head. () represents the Softmax function, where D is the number of channels per header. ;

[0104] The outputs of all heads are concatenated and then fused with time information through a fully connected layer.

[0105] ,

[0106] in, Features after fusing time information, For feature splicing processing, , For learnable weight matrix, It is the bias vector;

[0107] Finally, the output is reconstructed to the original spatial dimension to obtain the temporal output features extracted by the temporal attention mechanism. Its expression is:

[0108] ,

[0109] 2. Spatial Attention Module:

[0110] Spatial attention uses a multi-head attention mechanism to construct a non-local spatial dependency framework, enabling the model to adaptively focus on emotionally abnormal association regions with long-range features on the surface. The specific implementation steps are as follows.

[0111] Step 1: Input Feature Reconstruction and Location Encoding

[0112] The feature data is subjected to spatial feature reconstruction processing; the two-dimensional spatial location code is decomposed into horizontal direction code and vertical direction code, and a learnable spatial location code is generated by adding the outer product; the learnable spatial location code is added to the input tensor after feature reconstruction processing, and the face RGB and depth information features with spatial location code are output.

[0113] Specifically, each type of feature includes the following processing steps:

[0114] Input tensor Time series length Merge into batch dimension and spatial dimension Merging makes the features become This facilitates parallel computation on spatial sequences; the two-dimensional spatial location code is decomposed into horizontal and vertical codes, and a learnable spatial location code is generated by adding the outer products to record spatial feature location information;

[0115] ,

[0116] in, For learnable spatial location encoding, Encoding in the horizontal direction, Encoding in the vertical direction, They respectively indicate that it is a length of or A vector of all 1s It represents the Kronecker product.

[0117] Input after dimensional transformation Add learnable spatial location coding By aligning dimensions through a broadcast mechanism, spatial location awareness is enhanced, resulting in processed spatial input features. Its expression is:

[0118] ,

[0119] The second step is spatial multi-head self-attention calculation:

[0120] A query is generated from the spatially encoded RGB and depth features of a face through three independent linear transformations. ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get Let Q, K, and V represent the query, key, and value of the Nth head, respectively, with each head focusing on a different spatial pattern. Then, the spatial association weights of each head are independently calculated and weighted fused. The outputs of all heads are concatenated and then fused through a fully connected layer to integrate the spatial information. Finally, the output is reconstructed to the original spatial dimension to obtain the spatial output features extracted by the spatial attention mechanism. Specifically, each type of feature includes the following processing steps:

[0121] Association weights and weighted fusion calculation:

[0122] ,

[0123] in, The attention feature output for the Nth head. () represents the Softmax function, S is the spatial feature dimension, and D is the number of channels per head. ;

[0124] Fusion time information processing and calculation:

[0125] ,

[0126] in, To integrate spatial information and its features, For feature splicing processing, , For learnable weight matrix, It is the bias vector;

[0127] Spatial output characteristics calculate:

[0128] ,

[0129] 3. Channel Attention Module:

[0130] Channel attention incorporates a multi-head attention mechanism, focusing primarily on the feature importance allocation along the channel dimension, thus removing positional encoding modules that have less impact on them. Furthermore, to reduce parameter redundancy between channels, orthogonalization is used to initialize the 3D convolutional kernels. The specific implementation steps are as follows.

[0131] The first step is to enhance channel independence:

[0132] Input tensor 3D convolutional kernels initialized with orthogonality Redundant channel information is filtered to enhance channel independence, and compressed along the spatiotemporal dimension to extract the channel statistical feature tensor, whose expression is:

[0133] ,

[0134] ,

[0135] in, convolution kernel The filtered input tensor, where X is the input tensor. These represent the batch size, channel dimension, time series length, feature height, and width of the input features, respectively. For feature reconstruction operations, To achieve a mapping from an input of arbitrary size to an output of fixed size by performing weighted summation and scaling transformation on the input features, Input feature tensors into the processed channels;

[0136] The second step is to calculate the channel multi-head self-attention:

[0137] Channel input feature tensor Generate a query through three independent linear transformations ,key ,value ; will query ,key ,value Divided along the channel dimension The query retrieves the Nth head. Each head focuses on different channel modes; then, attention weights are calculated independently for each head; the attention weights of all heads are concatenated and then fused with channel information through a fully connected layer to obtain channel weights; finally, the channel weights are multiplied by the input tensor after generating a channel mask through an activation function to obtain the channel output features after feature importance allocation in the channel dimension.

[0138] Attention weight calculation:

[0139] ,

[0140] in, The attention feature output for the Nth head. () represents the Softmax function, C is the channel feature dimension, and D is the number of channels per head. ;

[0141] Channel weight calculation:

[0142] ,

[0143] in, The channel weights after integrating channel information, For feature splicing processing, , For learnable weight matrix, It is the bias vector;

[0144] Channel output characteristic calculation:

[0145] ,

[0146] in, For channel output characteristics, This is handled by the Sigmoid function.

[0147] 4. Multi-dimensional cross-gating fusion module

[0148] The fusion module employs a gating mechanism, using a dynamic weight allocation strategy to effectively integrate multi-dimensional features. This mechanism adaptively adjusts the weight coefficients of each information channel based on the statistical characteristics of the input features, preserving key features while suppressing redundant information, thereby mitigating potential conflicts between multimodal information. Furthermore, the gating structure, by introducing a nonlinear transformation function, not only enhances the interactive modeling capability between features of different dimensions but also avoids the over-propagation of duplicate features through an information flow regulation mechanism. The specific implementation steps are as follows.

[0149] Step 1, Dynamic Weight Learning:

[0150] The RGB and depth features of the face extracted by the temporal, spatial, and channel attention modules are respectively processed through convolutional layers to learn the weights of each dimension. The expression is as follows:

[0151]

[0152]

[0153] ,

[0154] in, The weights are for the time dimension, the spatial dimension, and the channel dimension, respectively. For the Sigmoid function, These are the gating weight matrix parameters for the time dimension, spatial dimension, and channel dimension, respectively. For time-based output features, For spatial output features, For channel output characteristics;

[0155] The second step is feature fusion:

[0156] The outputs from the three dimensions are gated and weighted, then superimposed, and obtained through residual connection and layer normalization to obtain the fused features of the temporal, spatial, and channel dimensions. Its expression is:

[0157] ;

[0158] in, The result after fusing the outputs from the three dimensions. Represents learnable parameters. It is a non-linear mapping function.

[0159] S3: The TSC-CGAM model outputs the emotion anomaly recognition result of the feature data.

[0160] Example 3

[0161] The difference between this embodiment and the previous embodiment is that it also includes S4:

[0162] S4: Extract one or more of the time output features, spatial output features, channel output features, and three-dimensional fusion features from the TSC-CGA unit, convert them into a visual heatmap, and overlay the visual heatmap onto the original face RGB image to achieve feature map visualization.

[0163] Furthermore, the conversion of the visualized heatmap includes the following steps:

[0164] Calculate the gradient weights of the features to be transformed; then... Feature maps of attention in each spatiotemporal channel are used This indicates that the gradient weights of the corresponding features are... It can be expressed by the following formula:

[0165] ,

[0166] in, The predicted score for target category c.

[0167] Calculate the 3D heatmap of the feature to be transformed based on the gradient weights; the corresponding 3D heatmap It can be expressed by the following formula:

[0168]

[0169] The 3D heatmap is compressed into a 2D heatmap, and the output is a visualized heatmap. The 3D heatmap is compressed into a 2D heatmap by taking the average value over the time dimension to facilitate two-dimensional visualization mapping.

[0170]

[0171] Furthermore, for a video sequence containing N frames, the frame-by-frame heatmap can be represented as:

[0172]

[0173] in, For the original image of frame t, This indicates an element-wise multiplication and addition operation. Upsample uses bilinear interpolation to achieve spatial alignment.

[0174] By employing the aforementioned spatiotemporal channel feature decoupling visualization method, a correlation mapping between the model's high-dimensional semantic features and dynamic changes in facial micro-expressions is constructed. Through this mapping mechanism, significant differences in facial features between individuals with emotional abnormalities and healthy individuals can be accurately captured, thereby effectively identifying feature changes that play a crucial role in the recognition of emotional abnormalities and deeply analyzing the root causes of these differences.

[0175] Example 4

[0176] This embodiment is a specific simulation example of the 3D face video emotion anomaly recognition method based on multidimensional cross-attention described in Embodiment 3. This embodiment takes offline emotion anomaly recognition as the application scenario and uses RGB-D face video data to illustrate the specific implementation process of the time-space-channel multidimensional decoupled attention architecture (TSC), multidimensional dynamic cross-gated fusion structure (TSC-CGA), and TSC-CGAM described in this invention.

[0177] 1. Data Acquisition and Preprocessing

[0178] A depth camera is used to simultaneously acquire RGB video sequences of faces and corresponding depth map sequences. Face detection is performed on the RGB frames, and the coordinates of the cropped and aligned face regions are synchronously applied to the depth map after affine transformation. The RGB images are then standardized, and the depth maps are normalized.

[0179] 2. TSC module processing

[0180] (1) Temporal attention submodule: In order to effectively capture the dynamic dependencies between frames in the video sequence, the temporal attention module first processes the video tensor containing RGB and depth information. Reorganized along spatial dimensions Feature sequence set; then superimposed with shared temporal location codes This process injects temporal sequence information into the feature sequence; utilizes a multi-head self-attention mechanism to calculate the similarity of each time step through query, key, and value vectors generated by linear transformation, further obtaining attention scores that characterize the importance between frames; finally, it preserves the original signal features through residual connections and layer normalization operations, outputting a feature tensor containing temporal dependencies. .

[0181] (2) Spatial Attention Submodule: The spatial attention module is used to mine long-range interactions in different facial regions. For the input tensor A shared two-dimensional location code is superimposed on the sequence, so that each pixel carries its specific coordinate information in the facial plane; then, multi-head self-attention is used to calculate the correlation between any two points in the sequence, highlighting semantically similar regions; and the process ends with residual connections and layer normalization to generate a spatially weighted feature map. .

[0182] (3) Channel Attention Submodule: To achieve adaptive weighting of RGB and depth channels, Gram-Schmidt orthogonalization is first applied to a set of point convolutional kernel weights to generate uncorrelated channel feature maps; then, global pooling is performed on the maps to compress spatial and temporal information into channel descriptions; finally, the multi-head self-attention sublayer is input to capture the complementarity and redundancy between channels, and channel weights are generated by Sigmoid activation to dynamically scale the original channels, thereby realizing the feature tensor that integrates RGB and depth information. .

[0183] 3. TSC-CGA Three-Dimensional Cross-Gated Fusion

[0184] In obtaining timing ,space ,aisle After establishing three independent attention vectors, a cross-gating mechanism is used to organically fuse the three types of information to fully exploit the complementary information between the modalities. Corresponding gating features are constructed for each pair of attention vectors. This feature determines the weight of the corresponding dimension's influence on another dimension during fusion; then, the three types of attention vectors are weighted and combined to generate a unified fusion attention mask; finally, this mask is applied again to the previously weighted feature map through a residual connection to obtain the final fusion feature tensor. .

[0185] 4. TSC-CGAM Network Structure

[0186] Insert TSC-CGA modules before and after each residual unit in the ResNet-18 backbone network, such as... Figure 2 As shown, this is used to further extract deeper spatiotemporal texture information. The network input is a four-channel tensor formed by concatenating the RGB and Depth channels. After the residual blocks are stacked, a fully connected layer is connected to the classification head to output the emotional abnormality score.

[0187] 5. Emotion Anomaly Recognition Results: The proposed method (TSC-CGAM) is compared with two baseline methods. This embodiment uses accuracy, precision, recall, F1 score, and area under the ROC curve (AUC) as evaluation metrics to analyze the model's performance in emotion anomaly recognition. As shown in Table 1, the proposed TSC-CGAM outperforms the two baseline methods in most metrics, particularly improving accuracy and AUC by approximately 8% and 0.07 respectively, demonstrating the effectiveness of the multidimensional decoupled attention and dynamic cross-gating fusion structure in emotion anomaly recognition tasks.

[0188] Table 1. Performance comparison of each method on the collected test data set.

[0189]

[0190] Example 5

[0191] like Figure 4 As shown, a 3D face video emotion anomaly recognition device based on multidimensional cross-attention includes at least one processor, a memory communicatively connected to the at least one processor, and at least one input / output interface communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enables the at least one processor to perform the multidimensional cross-attention 3D face video emotion anomaly recognition method described in the foregoing embodiments. The input / output interface may include a display, keyboard, mouse, and USB interface for inputting and outputting data.

[0192] Furthermore, the multidimensional cross-attention 3D face video emotion anomaly recognition device can be a desktop computer, mobile phone, tablet computer, wearable multidimensional cross-attention 3D face video emotion anomaly recognition device, etc., capable of performing deep information recognition.

[0193] Furthermore, the processor may include one or more processing cores. The processor connects various parts of the multi-dimensional cross-attention 3D face video emotion anomaly recognition device using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also be implemented separately as a communication chip, without being integrated into the processor.

[0194] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets, such as instructions or code sets used to implement the multi-dimensional cross-attention 3D face video emotion anomaly recognition method provided in this application embodiment. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created during the use of the multi-dimensional cross-attention 3D face video emotion anomaly recognition device (such as a modulation sequence-depth mapping table, image data, spectrogram data, etc.).

[0195] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0196] When the integrated units of the present invention are implemented as software functional units and sold or used as independent products, they can also be stored in a computer-readable storage medium. The computer-readable storage medium stores program code, which can be called by a processor to execute the methods described in the above method embodiments. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that executes any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0197] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for emotion anomaly recognition in 3D face videos based on multidimensional cross-attention, characterized in that, Includes the following steps: S1: Acquire the face data to be analyzed, perform preprocessing, and output feature data; The face data to be analyzed includes the face video to be analyzed and the corresponding depth information, and the feature data includes face RGB features and depth information features; The preprocessing includes face region cropping, face RGB and depth information alignment, feature normalization, and feature extraction processing. The output is RGB-D data, which is then input into a pre-constructed convolutional neural network for feature extraction, outputting face RGB features and corresponding depth information features. S2: Input the feature data into the pre-trained TSC-CGAM model; S3: The TSC-CGAM model outputs the emotion anomaly recognition result of the feature data; The TSC-CGAM model uses a convolutional neural network model as its backbone structure, and inserts a TSC-CGA unit after each feature extraction unit in the convolutional neural network model; the TSC-CGA unit includes a temporal attention module, a spatial attention module, a channel attention module, and a multidimensional cross-gated fusion module. The multi-dimensional cross-gating fusion module includes the following operation process: The RGB and depth features of the face extracted by the temporal attention module, spatial attention module, and channel attention module are respectively learned by convolutional layers to learn the weights of each dimension. The outputs of the three modules are then gated and weighted and superimposed, and then normalized by residual connections to obtain the fused features of temporal, spatial, and channel dimensions.

2. The method for identifying emotional anomalies in 3D face videos based on multidimensional cross-attention as described in claim 1, characterized in that, The time-conscious module includes the following operation flow: The feature data is subjected to temporal feature reconstruction processing; a learnable one-dimensional temporal location code is added to the input tensor after feature reconstruction processing, and the face RGB features and depth information features with temporal location code are output. The query is generated by performing three independent linear transformations on the RGB features and depth information features of the face with time location encoding. ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get Each head focuses on a different time pattern; Next, based on the temporal correlation of each head, the weights associated with each head are obtained and weighted fusion is performed; the outputs of all heads are concatenated and then fused with temporal information through a fully connected layer; finally, the outputs are reconstructed to the original spatial dimension to obtain the temporal output features extracted by the temporal attention mechanism; the expression for calculating the temporal output features is as follows: Calculate the association weights and perform weighted fusion: ， Calculation of time-related information features: ， Time output feature calculation: ， in, The attention feature output for the Nth head. () is the activation function. express Matrix transpose These represent the batch size, channel dimension, time series length, feature height, and width of the input features, respectively, where D is the number of channels for each head. Features after fusing time information, For feature splicing processing, , For learnable weight matrix, For bias vectors, For feature reconstruction operations, This is a time-based output feature.

3. The method for emotion anomaly recognition in 3D face videos based on multidimensional cross-attention as described in claim 1, characterized in that, The spatial attention module includes the following operation flow: The feature data is subjected to spatial feature reconstruction processing; the two-dimensional spatial location code is decomposed into horizontal and vertical direction codes, and a learnable spatial location code is generated by adding them through an outer product; the learnable spatial location code is added to the input tensor after feature reconstruction processing, and the output is a face RGB and depth information feature with spatial location code; the expression of the learnable spatial location code is: ， in, For learnable spatial location encoding, Encoding in the horizontal direction, Encoding in the vertical direction, They respectively indicate that it is a length of or A vector of all 1s Indicates the Kronecker product; A query is generated from the spatially encoded RGB and depth features of a face through three independent linear transformations. ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get Each head focuses on different spatial patterns; then, the correlation weights in the spatial dimension of each head are independently calculated and weighted fusion is performed; the outputs of all heads are concatenated and then fused through a fully connected layer to integrate spatial information; finally, the output is reconstructed to the original spatial dimension to obtain the spatial output features extracted by the spatial attention mechanism; the expression for the calculation process of the spatial output features is as follows: Calculate the association weights and perform weighted fusion: ， Fusion spatial information feature calculation: ， Spatial output feature calculation: ， in, The attention feature output for the Nth head. () is the activation function. express Matrix transpose These represent the batch size, channel dimension, time series length, feature height, and width of the input features, respectively, where D is the number of channels for each head. , To integrate spatial information and its features, For feature splicing processing, , For learnable weight matrix, Let S be the bias vector, and S be the spatial feature dimension. For spatial output features, This is a feature reconstruction operation.

4. The method for emotion anomaly recognition in 3D face videos based on multidimensional cross-attention as described in claim 1, characterized in that, The channel attention module includes the following operation flow: The feature data is processed through orthogonally initialized 3D convolution kernels. Redundant channel information is filtered out and compressed along the spatiotemporal dimension to extract the channel statistical feature tensor, whose expression is: ， ， in, convolution kernel The filtered input tensor, where X is the input tensor. To achieve a mapping from an input of arbitrary size to an output of fixed size by performing weighted summation and scaling transformation on the input features, Input the feature tensor to the processed channel. For feature reconstruction operations; The channel statistical feature tensor generates a query through three independent linear transformations. ,key ,value ; will query ,key ,value Divided along the channel dimension Size, get Each head focuses on different channel modes; then, attention weights are calculated independently for each head; the attention weights of all heads are concatenated and fused with channel information through a fully connected layer to obtain channel weights; finally, the channel weights are multiplied by the input tensor after generating a channel mask through an activation function to obtain the channel output features after feature importance allocation in the channel dimension; the expression for the calculation process of the channel output features is as follows: Calculate the association weights and perform weighted fusion: ， Calculation of fused channel information features: ， Channel output characteristic calculation: ， in, The attention feature output for the Nth head. () represents the activation function, and D represents the number of channels in each head. , express Matrix transpose These represent the batch size, channel dimension, time series length, feature height, and width of the input features, respectively. The channel weights after integrating channel information, For feature splicing processing, , For learnable weight matrix, For bias vectors, For channel output characteristics, This is the Sigmoid activation function.

5. The method for identifying emotional anomalies in 3D face videos based on multidimensional cross-attention according to claim 1, characterized in that, The expression for the fusion feature of the three dimensions of time, space, and channel is: in, The result is the fusion of the outputs from the three dimensions, where θ represents the learnable parameters, and FusionFunc is a non-linear mapping function. For time-based output features, For spatial output features, This refers to the channel output characteristics.

6. The method for identifying emotional anomalies in 3D face videos based on multidimensional cross-attention according to claim 1, characterized in that, The TSC-CGAM model is trained using labeled face videos and depth information datasets. After convergence, the model is a TSC-CGAM model capable of recognizing emotional anomalies.

7. The method for emotion anomaly recognition in 3D face videos based on multidimensional cross-attention as described in claim 1, characterized in that, Also includes S4: S4: Extract one or more of the time output features, spatial output features, channel output features, and three-dimensional fusion features from the TSC-CGA unit, convert them into a visual heatmap, and overlay the visual heatmap onto the original face RGB image to achieve feature map visualization.

8. The method for emotion anomaly recognition in 3D face videos based on multidimensional cross-attention as described in claim 7, characterized in that, The conversion of the visualization heatmap includes the following steps: Calculate the gradient weights of the features to be transformed; Calculate the 3D heatmap of the feature to be converted based on the gradient weights; The 3D heatmap is compressed into a 2D heatmap and output as a visual heatmap.

9. A device for recognizing emotional anomalies in 3D facial videos based on multidimensional cross-attention, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform a method for identifying emotional anomalies in 3D face videos based on multidimensional cross-attention, as described in any one of claims 1 to 8.

Images