Method and device for recognizing emotion abnormity based on multi-dimensional cross attention 3D face video

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a method and equipment for recognizing emotion abnormity of a 3D face video based on multi-dimensional cross attention. According to the invention, through a proposed attention framework, RGB visual features and face depth features are deeply fused by using space-time channel three-dimensional features; and a long-distance and long-time sequence dependency relationship is captured through a multi-head self-attention mechanism in three dimensions, and dynamic weighted fusion of features of different dimensions is realized through a cross-gating mechanism, so that the discrimination of feature expression is effectively enhanced, and redundant feature interference is inhibited while information integrity is kept. Therefore, feature extraction of dynamic change of facial expression muscles is optimized, cross-modal complementary representation of RGB texture and depth geometric features is realized, and emotion anomaly recognition is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for identifying abnormal emotions in 3D facial videos based on multi-dimensional cross-attention. Background Art

[0002] Identifying emotional abnormalities can aid in the screening and early intervention of mental illness. Traditional methods rely primarily on psychological scales and clinical interviews, which are subject to high subjectivity, concealment, and time-consuming diagnostics. In recent years, AI-based facial video analysis technology has provided a new approach for objectively identifying emotional abnormalities.

[0003] However, existing methods mainly rely on 2D-RGB video data in terms of data, using only visual features, with insufficient three-dimensional dynamic perception, making it difficult to model the three-dimensional dynamic characteristics of facial muscle movements; in terms of modeling methods, most methods separate and extract spatiotemporal features, and perform modal fusion through feature splicing and weighted methods, ignoring cross-modal complementarity, making it difficult to characterize micro-expression transients and facial geometric deformations related to emotional abnormalities.

[0004] Therefore, there is a need for a method and device for identifying emotional anomalies in 3D facial videos based on multi-dimensional cross-attention, which can deeply mine cross-modal complementarity and have higher recognition accuracy. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned deficiencies in the prior art and to provide a method and device for identifying abnormal emotions in 3D facial videos based on multi-dimensional cross-attention.

[0006] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions: A method for identifying abnormal emotions in 3D face videos based on multi-dimensional cross-attention includes the following steps: S1: Acquire facial data to be analyzed, perform preprocessing, and output feature data; the facial data to be analyzed includes the facial video to be analyzed and the corresponding depth information, and the feature data includes facial RGB features and depth information features; S2: Input the feature data into the pre-trained TSC-CGAM model; S3: The TSC-CGAM model outputs an emotion anomaly recognition result of the feature data; Among them, the TSC-CGAM model adopts a convolutional neural network model as the backbone structure, and inserts a TSC-CGA unit after each feature extraction unit in the convolutional neural network model; the TSC-CGA unit includes a temporal attention module, a spatial attention module, a channel attention module and a multi-dimensional cross-gated fusion module.

[0007] As a preferred solution of the present invention, the preprocessing includes face area cropping, face RGB and depth information alignment, feature normalization and feature extraction processing.

[0008] As a preferred solution of the present invention, the time attention module includes the following operation process: Performing temporal feature reconstruction processing on the feature data; adding a learnable one-dimensional temporal position code to the input tensor after the feature reconstruction processing, and outputting the RGB features and depth information features of the face with the temporal position code; The face RGB features and depth information features with temporal position encoding are transformed into query by three independent linear transformations ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get , each head focuses on different time patterns; then, based on the correlation between each head in the time dimension, the weight associated with each head is obtained and weighted fusion is performed; the outputs of all heads are spliced ​​and the time information is fused through the fully connected layer; finally, the output is reconstructed to the original spatial dimension to obtain the time output features extracted by the time attention mechanism; the calculation process expression of the time output feature is: Calculate the association weights and perform weighted fusion: , Fusion time information feature calculation: , Time output feature calculation: , in, is the attention feature output of the Nth head, () is the activation function, They represent the batch size, channel dimension, time series length, feature height and width of the input features respectively, and D is the number of channels of each head. is the feature after integrating time information, For feature splicing processing, , is the learnable weight matrix, is the bias vector, is the feature reconstruction operation, Output features for time.

[0009] As a preferred solution of the present invention, the spatial attention module includes the following operation process: The feature data is subjected to spatial feature reconstruction processing; the two-dimensional spatial position code is decomposed into a horizontal direction code and a vertical direction code, and a learnable spatial position code is generated by adding outer products; the learnable spatial position code is added to the input tensor after the feature reconstruction processing, and the RGB and depth information features of the face with the spatial position code are output; the expression of the learnable spatial position code is: , in, is a learnable spatial position encoding, For horizontal coding, For vertical coding, It indicates that its length is or A vector of all 1s, represents the Kronecker product

[0010] The facial RGB and depth information features with spatial position encoding are transformed into queries through three independent linear transformations. ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get , each head focuses on different spatial patterns; then the association weights in the spatial dimension are calculated independently for each head and weighted fusion is performed; the outputs of all heads are concatenated and the spatial information is fused through a fully connected layer; finally, the output is reconstructed to the original spatial dimension to obtain the spatial output features extracted by the spatial attention mechanism; the calculation process of the spatial output features is expressed as: Calculate the association weights and perform weighted fusion: , Fusion spatial information feature calculation: , Spatial output feature calculation: , in, is the attention feature output of the Nth head, S is the spatial feature dimension, and D is the number of channels of each head. , is the feature after integrating spatial information, For feature splicing processing, , is the learnable weight matrix, is the bias vector, Output features for the space.

[0011] As a preferred solution of the present invention, the channel attention module includes the following operation process: The feature data is orthogonally initialized with a 3D convolution kernel Filter out redundant channel information, compress along the spatiotemporal dimension, and extract the channel statistical feature tensor, which is expressed as: , , in, is the convolution kernel The filtered input tensor, X is the input tensor, In order to achieve the mapping from arbitrary size input to fixed size output by weighted summation and scale transformation of input features, Input feature tensor for the processed channels; The channel statistical feature tensor is transformed into a query through three independent linear transformations ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get , each head focuses on different channel modes; then the attention weight is calculated independently for each head; the attention weights of all heads are spliced ​​together and the channel information is fused through the fully connected layer to obtain the channel weight; finally, the channel weight is generated by the activation function. The channel mask is multiplied by the input tensor to obtain the channel output feature after the feature importance is distributed in the channel dimension; the calculation process expression of the channel output feature is: Calculate the association weights and perform weighted fusion: , Fusion channel information feature calculation: , Channel output feature calculation: , in, is the attention feature output of the Nth head, C is the channel feature dimension, and D is the number of channels for each head. , is the channel weight after fusing channel information, For feature splicing processing, , is the learnable weight matrix, is the bias vector, is the channel output feature, is the Sigmoid activation function.

[0012] As a preferred embodiment of the present invention, the multi-dimensional cross-gated fusion module includes the following operation process: The RGB features and depth information features of the face extracted by the time attention module, space attention module and channel attention module are respectively passed through the convolution layer to learn the weights of each dimension; the outputs of the three modules are superimposed after gated weighting, and are normalized through residual connection and layer normalization to obtain the three-dimensional fusion features of time, space and channel. ; The expression of the three-dimensional fusion feature of time, space and channel is:

[0013] in, is the result of fusing the outputs of the three dimensions, θ represents the learnable parameters, and FusionFunc is a nonlinear mapping function. is the temporal output feature, is the spatial output feature, Output features for the channel.

[0014] As a preferred solution of the present invention, the TSC-CGAM model is trained by using labeled face videos and depth information datasets, and after model convergence, a TSC-CGAM model capable of recognizing emotional anomalies is obtained.

[0015] As a preferred embodiment of the present invention, the present invention further comprises S4: S4: Extract one or more of the temporal output features, spatial output features, channel output features, and three-dimensional fusion features output by the TSC-CGA unit, convert them into a visual heat map, and superimpose the visual heat map on the original face RGB image to achieve feature map visualization.

[0016] As a preferred embodiment of the present invention, the conversion of the visualized heat map includes the following steps: Calculate the gradient weight of the feature to be converted; Calculate a 3D heat map of the feature to be converted according to the gradient weight; The 3D heat map is compressed into a 2D heat map and output as a visual heat map.

[0017] A device for identifying emotional anomalies in 3D facial videos based on multi-dimensional cross-attention, comprising at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned methods for identifying emotional anomalies in 3D facial videos based on multi-dimensional cross-attention.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This paper proposes an attention framework that deeply fuses RGB visual features with facial depth features using the three-dimensional features of spatiotemporal channels. Furthermore, a multi-head self-attention mechanism is used to capture long-range and long-term dependencies in three dimensions. A cross-gating mechanism is used to dynamically weightedly fuse features from different dimensions, effectively enhancing the discriminability of feature representations while maintaining information integrity and suppressing redundant feature interference. This optimizes feature extraction of dynamic facial expression muscles, achieving cross-modal complementary representation of RGB texture and deep geometric features, and significantly improving the accuracy of identifying abnormal emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a method for identifying abnormal emotions in 3D facial videos based on multi-dimensional cross-attention according to Example 1 of the present invention; Figure 2 This is a schematic diagram of the structure of the TSC-CGAM model in the method for identifying abnormal emotion in 3D facial videos based on multi-dimensional cross-attention according to Example 2 of the present invention; Figure 3 This is a schematic diagram of the structure of the TSC-CGA unit in the method for identifying abnormal emotion in 3D facial videos based on multi-dimensional cross-attention according to Example 2 of the present invention; Figure 4 This is a structural schematic diagram of a device for identifying anomaly emotions in 3D facial videos based on multi-dimensional cross-attention, which utilizes a method for identifying anomaly emotions in 3D facial videos based on multi-dimensional cross-attention as described in the previous embodiment, as described in Example 5 of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described in detail below in conjunction with test examples and specific embodiments. However, this should not be understood as limiting the scope of the present invention to the following embodiments, and all technologies implemented based on the present invention fall within the scope of the present invention.

[0021] Example 1 like Figure 1 As shown, a method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention includes the following steps: S1: Obtain the facial data to be analyzed, perform preprocessing, and output feature data; the facial data to be analyzed includes the facial video to be analyzed and the corresponding depth information, and the feature data includes facial RGB features and depth information features.

[0022] S2: Input the feature data into the pre-trained TSC-CGAM model (Temporal-Spatial-Channel Cross-Gated Attention Model).

[0023] S3: The TSC-CGAM model outputs an emotion anomaly recognition result of the feature data.

[0024] The TSC-CGAM model adopts a convolutional neural network model as its backbone structure, and inserts a TSC-CGA unit after each feature extraction unit in the convolutional neural network model; the TSC-CGA (Temporal-Spatial-Channel Cross-Gated Attention) unit includes a temporal attention module, a spatial attention module, a channel attention module, and a multi-dimensional cross-gated fusion module.

[0025] Example 2 This embodiment is a specific implementation of the method for identifying abnormal emotions in 3D facial videos based on multi-dimensional cross-attention described in Example 1, including the following steps: S1: Obtain the facial data to be analyzed, perform preprocessing, and output feature data.

[0026] The preprocessing includes face area cropping, face RGB and depth information alignment, feature normalization and feature extraction processing; the face data to be analyzed includes the face video to be analyzed and the corresponding depth information, and the feature data includes face RGB features and depth information features.

[0027] Furthermore, in this step, a 3D camera is used to synchronously capture the RGB video stream and facial depth information during acquisition, thereby acquiring a three-dimensional facial video of the subject. The depth data can quantify three-dimensional dynamic features such as nasolabial groove deformation and zygomatic muscle contraction amplitude, thus making up for the shortcomings of traditional 2D data in geometric motion modeling.

[0028] In this embodiment, after acquiring a facial video, facial key points are first detected. The detected facial region is aligned with the depth information image, and the image is uniformly captured at a preset size. To ensure that the data dimensions of the facial depth image are consistent with those of the facial video, normalization is performed and the output is RGB-D data (i.e., the facial RGB image and depth information). This RGB-D data is then input into a pre-built convolutional neural network for feature extraction, outputting facial RGB features and corresponding depth information features.

[0029] S2: Input the feature data into the pre-trained TSC-CGAM model.

[0030] The TSC-CGAM model captures the temporal dynamics and spatial distribution of facial micro-expressions through spatiotemporal attention and fuses RGB visual features with facial depth information using channel attention. The spatiotemporal and channel-wise attention mechanisms are fused through a dynamic gating strategy to exploit the nonlinear complementary relationship between RGB and depth features in videos.

[0031] Furthermore, the convolutional neural network in the TSC-CGAM model in this embodiment can adopt a ResNet model, a VGG model, or a DenseNet model. For example, when the convolutional neural network is a ResNet model, the TSC-CGA unit is inserted before and after each residual module. When the convolutional neural network is a VGG model, the TSC-CGA unit is inserted before each maximum pooling layer (MaxPooling). When the convolutional neural network is a DenseNet model, the TSC-CGA unit is inserted before each conversion layer.

[0032] Further, such as Figure 2 As shown in the figure, in this embodiment, the TSC-CGAM model is a ResNet model. The model inputs feature data, and the TSC-CGA unit replaces the bottleneck layer in the intermediate layer, generating a weighted feature with deep fusion of multi-dimensional features. After passing through multiple residual modules, the adaptive pooling layer resamples the features to a fixed shape. Finally, a fully connected layer predicts the sentiment anomaly score as the final output of the TSC-CGAM.

[0033] Furthermore, the TSC-CGAM model is trained using a dataset of labeled face videos and depth information. After model convergence, a TSC-CGAM model capable of identifying emotional anomalies is obtained. The video dataset includes the face videos to be analyzed and the corresponding depth information. Specifically, model training includes the following steps: a: Analyze the face video and the corresponding depth information and perform preprocessing; b: Inputting the pre-processed face video to be analyzed and the depth information into a pre-built convolutional neural network for feature extraction, and outputting the face RGB features and the corresponding depth information features; c: Input the facial RGB features and depth information features into the pre-built TSC-CGA multi-dimensional cross-gated attention mechanism unit to fuse the facial RGB features and depth information features; The TSC-CGA unit includes a temporal attention module, a spatial attention module, a channel attention module, and a multi-dimensional cross-gating fusion module. It extracts temporal features, spatial features, and channel features corresponding to facial RGB features and depth information features, and inputs them into the multi-dimensional cross-gating module for feature fusion.

[0034] d: The fusion features output by the TSC-CGA multi-dimensional cross-gated attention mechanism unit are fed into the fully connected layer to output the final emotion anomaly recognition result.

[0035] Furthermore, it also includes training the TSC-CGAM model based on the degree of emotional abnormality of different users and adjusting the parameters of the TSC-CGAM model.

[0036] e: Obtain the trained TSC-CGAM model; extract the temporal features, spatial features, channel features of the TSC-CGA multi-dimensional cross-gated attention mechanism unit of the TSC-CGAM model, and the features after multi-dimensional cross-gating fusion for feature map visualization.

[0037] In the TSC-CGA unit, the spatial attention module, the temporal attention module, and the channel attention module generate internal features and attention vectors, such as Figure 3 shown.

[0038] 1. Time Attention Module: Temporal attention uses a multi-head attention mechanism, combining the advantages of the Transformer architecture to build a long-term dependency modeling framework, automatically focusing on key frames with drastic expression changes. The specific implementation steps are as follows: Step 1: Input feature reconstruction and position encoding: Performing temporal feature reconstruction processing on the feature data; adding a learnable one-dimensional temporal position code to the input tensor after the feature reconstruction processing, and outputting the RGB features and depth information features of the face with the temporal position code; Furthermore, this embodiment will input tensor spatial dimension Merge into the batch dimension, so that the features become , which facilitates parallel computing on time series; Represents the batch size, channel dimension, time series length, feature height and width of the input features respectively; the input of the transformed dimension in the feature reconstruction process Add learnable one-dimensional temporal position encoding , to save and record the position of each time frame, align the dimensions through the broadcast mechanism, enhance the temporal position perception, and obtain the processed time input features ; Its expression is: , The second step is temporal multi-head self-attention calculation: The face RGB features and depth information features with temporal position encoding are transformed into query by three independent linear transformations ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get , each head focuses on different time patterns; then, according to the correlation between each head in the time dimension, the weight associated with each head is obtained and weighted fusion is performed; the outputs of all heads are spliced ​​together and the time information is fused through the fully connected layer; finally, the output is reconstructed to the original spatial dimension to obtain the temporal output features extracted by the temporal attention mechanism.

[0039] Specifically, each feature type includes the following processing steps: Time input characteristics The query is generated by three independent linear transformations ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get To focus on different time modes; then calculate the association weight in the time dimension for each head independently and perform weighted fusion; , in, is the attention feature output of the Nth head, () is the Softmax function processing, D is the number of channels of each head ; The outputs of all heads are concatenated and then the time information is fused through the fully connected layer; , in, is the feature after integrating time information, For feature splicing processing, , is the learnable weight matrix, is the bias vector; Finally, the output is reconstructed to the original spatial dimension to obtain the temporal output features extracted by the temporal attention mechanism ; Its expression is: , 2. Spatial Attention Module: Spatial attention uses a multi-head attention mechanism to build a non-local spatial dependency framework, enabling the model to adaptively focus on emotion-related abnormality areas of long-range facial features. The following are the specific implementation steps.

[0040] The first step is input feature reconstruction and position encoding: The feature data is subjected to spatial feature reconstruction processing; the two-dimensional spatial position code is decomposed into a horizontal direction code and a vertical direction code, and a learnable spatial position code is generated by outer product addition; the learnable spatial position code is added to the input tensor after the feature reconstruction processing, and the RGB and depth information features of the face with the spatial position code are output.

[0041] Specifically, each feature type includes the following processing steps: The input tensor The length of the time series Merge to the batch dimension and the spatial dimension Merge, so that the features become , which facilitates parallel computation on spatial sequences; decomposes the two-dimensional spatial position code into horizontal direction code and vertical direction code, and generates a learnable spatial position code by adding outer products to record spatial feature position information; , in, is a learnable spatial position encoding, For horizontal coding, For vertical coding, It indicates that its length is or A vector of all 1s, represents the Kronecker product.

[0042] Input after dimension transformation Adding learnable spatial position encoding , align dimensions through the broadcast mechanism, enhance spatial position perception, and obtain processed spatial input features ; Its expression is: , The second step is spatial multi-head self-attention calculation: The facial RGB and depth information features with spatial position encoding are transformed into queries through three independent linear transformations. ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get The query Q, key K, and value V of the Nth head are represented respectively, and each head focuses on different spatial patterns. The association weights in the spatial dimension are then calculated independently for each head and weighted fusion is performed. The outputs of all heads are concatenated and the spatial information is fused through a fully connected layer. Finally, the output is reconstructed to the original spatial dimension to obtain the spatial output features extracted by the spatial attention mechanism. Specifically, each type of feature includes the following processing steps: Association weight and weighted fusion calculation: , in, is the attention feature output of the Nth head, () is the Softmax function processing, S is the spatial feature dimension, and D is the number of channels for each head. ; Fusion time information processing calculation: , in, is the feature after integrating spatial information, For feature splicing processing, , is the learnable weight matrix, is the bias vector; Spatial output features calculate: , 3. Channel Attention Module: Channel attention incorporates a multi-head attention mechanism, focusing on the distribution of feature importance along the channel dimension, thereby removing the positional encoding module, which has a less significant impact on it. Furthermore, to reduce parameter redundancy across channels, orthogonal initialization of the 3D convolution kernel is used. The following are the specific implementation steps.

[0043] The first step is to enhance channel independence: The input tensor 3D convolution kernel after orthogonal initialization Redundant channel information is filtered to enhance channel independence, and compressed along the spatiotemporal dimension to extract the channel statistical feature tensor, which is expressed as: , , in, is the convolution kernel The filtered input tensor, X is the input tensor, Represent the batch size, channel dimension, time series length, feature height and width of the input features respectively, is the feature reconstruction operation, In order to achieve the mapping from arbitrary size input to fixed size output by weighted summation and scale transformation of input features, Input feature tensor for the processed channels; The second step is channel multi-head self-attention calculation: Channel input feature tensor The query is generated by three independent linear transformations ,key ,value ; will query ,key ,value Split along the channel dimension into Heads, get the query of the Nth head , each head focuses on different channel modes; then the attention weight is calculated independently for each head; the attention weights of all heads are spliced ​​together and the channel information is fused through the fully connected layer to obtain the channel weight; finally, the channel weight is generated by the activation function. The channel mask is multiplied by the input tensor to obtain the channel output feature after the feature importance is distributed in the channel dimension; Attention weight calculation: , in, is the attention feature output of the Nth head, () is the Softmax function processing, C is the channel feature dimension, and D is the number of channels for each head. ; Channel weight calculation: , in, is the channel weight after fusing channel information, For feature splicing processing, , is the learnable weight matrix, is the bias vector; Channel output feature calculation: , in, is the channel output feature, Processed by Sigmoid function.

[0044] 4. Multi-dimensional cross-gating fusion module The fusion module employs a gating mechanism, using a dynamic weight allocation strategy to effectively integrate multidimensional features. This mechanism adaptively adjusts the weight coefficients of each information channel based on the statistical properties of the input features, preserving key features while suppressing redundant information, thereby mitigating potential conflicts between multimodal information. Furthermore, by introducing a nonlinear transformation function, the gating structure not only enhances the ability to model interactions between features of different dimensions but also prevents the excessive propagation of duplicate features through information flow regulation. The following are the specific implementation steps.

[0045] The first step is dynamic weight learning: The RGB features and depth information features of the face extracted by the time attention module, spatial attention module, and channel attention module are respectively passed through the convolutional layer to learn the weights of each dimension, and the expression is:

[0046]

[0047] , in, are the weights of the time dimension, space dimension, and channel dimension respectively, is the Sigmoid function, are the gating weight matrix parameters of the time dimension, space dimension, and channel dimension respectively, is the temporal output feature, is the spatial output feature, Output features for the channel; The second step is feature fusion: The outputs of the three dimensions are gated and weighted and then superimposed, and the three-dimensional fusion features of time, space and channels are obtained through residual connection and layer normalization. ; Its expression is: ; in, The result after fusion of the three-dimensional output is: represents the learnable parameters, It is a nonlinear mapping function.

[0048] S3: The TSC-CGAM model outputs an emotion anomaly recognition result of the feature data.

[0049] Example 3 The difference between this embodiment and the above embodiment is that it further includes S4: S4: Extract one or more of the temporal output features, spatial output features, channel output features, and three-dimensional fusion features output by the TSC-CGA unit, convert them into a visual heat map, and superimpose the visual heat map on the original face RGB image to achieve feature map visualization.

[0050] Furthermore, the conversion of the visual heat map includes the following steps: Calculate the gradient weight of the feature to be converted; The feature map of spatiotemporal channel attention is used Represents, then the gradient weight of the corresponding feature It can be expressed by the following formula: , in, is the predicted score of target category c.

[0051] Calculate the 3D heat map of the feature to be converted according to the gradient weight; the corresponding 3D heat map It can be expressed by the following formula:

[0052] The 3D heat map is compressed into a 2D heat map and output as a visual heat map. By taking the average value in the time dimension, the 3D heat map is compressed into a 2D heat map to facilitate two-dimensional visual mapping:

[0053] Furthermore, for a video sequence containing N frames, the frame-by-frame heat map can be expressed as:

[0054] in, is the original image of frame t, Represents element-by-element multiplication and addition operations, and Upsample uses bilinear interpolation to achieve spatial alignment.

[0055] Using the above-mentioned decoupling and visualization method for spatiotemporal channel features, we construct a correlation mapping between the model's high-dimensional semantic features and the dynamic changes in facial micro-expressions. This mapping mechanism accurately captures the significant differences in facial features between people with abnormal emotions and healthy individuals, effectively identifying the characteristic changes that are critical for identifying abnormal emotions and deeply analyzing the root causes of these differences.

[0056] Example 4 This embodiment is a specific simulation example of the method for identifying abnormal emotions in 3D facial videos based on multi-dimensional cross-attention described in Example 3. This embodiment takes offline abnormal emotions recognition as the application scenario and uses RGB-D facial video data to illustrate the specific implementation process of the time-space-channel multi-dimensional decoupled attention architecture (TSC), multi-dimensional dynamic cross-gated fusion structure (TSC-CGA) and TSC-CGAM described in the present invention.

[0057] 1. Data Collection and Preprocessing A depth camera is used to synchronously capture facial RGB video sequences and corresponding depth map sequences. Face detection is performed on the RGB frames, and an affine transformation is used to crop and align the facial region, and the resulting coordinates are then applied to the depth map. The RGB images are normalized, and the depth map is normalized.

[0058] 2.TSC module processing (1) Temporal attention submodule: To effectively capture the dynamic dependencies between frames in a video sequence, the temporal attention module first converts the video tensor containing RGB and depth information into Reorganize along the spatial dimension into A collection of feature sequences; then superimposing shared temporal position codes , injecting temporal order information into the feature sequence; using the multi-head self-attention mechanism, the query, key, and value vectors generated by linear transformation are used to calculate the similarity of each time step, and further obtain the attention score that represents the importance between frames; finally, the original signal features are retained through residual connection and layer normalization operations, and the feature tensor containing temporal dependency is output. .

[0059] (2) Spatial attention submodule: The spatial attention module is used to explore the long-range interactions between different facial regions. , a shared two-dimensional position code is superimposed on the sequence, so that each pixel carries its specific coordinate information in the facial plane; then the multi-head self-attention is used to calculate the degree of association between any two points in the sequence, highlighting semantically similar areas; and finally, residual connection and layer normalization are performed to generate a spatially weighted feature map. .

[0060] (3) Channel attention submodule: To achieve adaptive weighting of RGB and depth channels, a set of point convolution kernel weights are first Gram–Schmidt orthogonalized to generate independent channel feature maps; then the maps are globally pooled to compress spatial and temporal information into channel descriptions; finally, the multi-head self-attention sublayer is input to capture the complementarity and redundancy between channels, and channel weights are generated through Sigmoid activation, and the original channels are dynamically scaled to achieve a feature tensor that integrates RGB and depth information. .

[0061] 3.TSC-CGA three-dimensional cross-gating fusion In obtaining the timing ,space ,aisle After three independent attention vectors are generated, in order to fully explore the complementary information between the modalities, the three types of information are organically integrated through the cross-gating mechanism. Corresponding gating features are constructed for each pair of attention. , which determines the weight of the influence of the corresponding dimension on the other dimension in the fusion; then the three types of attention vectors are weighted and combined to generate a unified fusion attention mask; finally, the mask is applied to the previously weighted feature map again through the residual connection to obtain the final fusion feature tensor .

[0062] 4.TSC-CGAM Network Structure Insert the TSC-CGA module before and after each residual unit in the ResNet-18 backbone network, such as Figure 2 As shown, in order to further extract deeper spatiotemporal texture information. The network input is a four-channel tensor composed of RGB and Depth channels. After the residual blocks are stacked, the classification head is connected to a fully connected layer to output the sentiment anomaly score.

[0063] 5. Results of Abnormal Emotion Recognition: This method (TSC-CGAM) was compared with the following two baseline methods. This example uses accuracy, precision, recall, F1-score, and area under the receiver operating characteristic (ROC) curve (AUC) as model output evaluation metrics to analyze the model's performance in abnormal emotion recognition. As shown in Table 1, the proposed TSC-CGAM outperforms both baseline methods in most metrics, particularly improving accuracy by approximately 8% and AUC by 0.07, respectively. This demonstrates the effectiveness of the multi-dimensional decoupled attention and dynamic cross-gating fusion architecture in the abnormal emotion recognition task.

[0064] Table 1 Performance comparison of various methods on the collected data test set

[0065] Example 5 like Figure 4As shown, a device for identifying abnormal emotions in 3D facial videos based on multi-dimensional cross-attention includes at least one processor, a memory in communication with the at least one processor, and at least one input / output interface in communication with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for identifying abnormal emotions in 3D facial videos based on multi-dimensional cross-attention as described in the aforementioned embodiment. The input / output interface may include a display, a keyboard, a mouse, and a USB interface for inputting and outputting data.

[0066] Furthermore, the multi-dimensional cross-attention 3D face video emotion anomaly recognition device can be a desktop computer, mobile phone, tablet computer, wearable multi-dimensional cross-attention 3D face video emotion anomaly recognition device, etc., which can perform depth information recognition.

[0067] Furthermore, the processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the multi-dimensional cross-attention 3D facial video emotion anomaly recognition device. By running or executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory, the processor performs various functions and processes data within the multi-dimensional cross-attention 3D facial video emotion anomaly recognition device. Optionally, the processor may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a communications chip.

[0068] The memory may include a random access memory (RAM) or a read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instruction sets, such as instructions or code sets for implementing a method for identifying abnormal emotions in 3D facial videos based on multi-dimensional cross-attention provided in an embodiment of the present application. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also be based on data created during use by the multi-dimensional cross-attention 3D facial video abnormal emotion recognition device (such as a mapping table of modulation sequence and depth, image data, spectrum graph data), etc.

[0069] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0070] When the integrated unit described above is implemented as a software functional unit and sold or used as a standalone product, it can also be stored in a computer-readable storage medium containing program code that can be invoked by a processor to execute the methods described in the above-mentioned method embodiments. Based on this understanding, the technical solutions of the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code for executing any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying abnormal emotions in 3D face videos based on multi-dimensional cross-attention, characterized by: The following steps are involved: S1: Acquire facial data to be analyzed, perform preprocessing, and output feature data; the facial data to be analyzed includes the facial video to be analyzed and the corresponding depth information, and the feature data includes facial RGB features and depth information features; S2: Input the feature data into the pre-trained TSC-CGAM model; S3: The TSC-CGAM model outputs an emotion anomaly recognition result of the feature data; Among them, the TSC-CGAM model adopts a convolutional neural network model as the backbone structure, and inserts a TSC-CGA unit after each feature extraction unit in the convolutional neural network model; the TSC-CGA unit includes a temporal attention module, a spatial attention module, a channel attention module and a multi-dimensional cross-gated fusion module.

2. The method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention according to claim 1, characterized in that: The preprocessing includes face area cropping, face RGB and depth information alignment, feature normalization and feature extraction processing.

3. The method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention according to claim 1, characterized in that: The time attention module includes the following operation process: Performing temporal feature reconstruction processing on the feature data; adding a learnable one-dimensional temporal position code to the input tensor after the feature reconstruction processing, and outputting the RGB features and depth information features of the face with the temporal position code; The face RGB features and depth information features with temporal position encoding are transformed into query by three independent linear transformations ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get ,Each head focuses on different temporal patterns; Then, based on the correlation between each head in the time dimension, the weight associated with each head is obtained and weighted fusion is performed; the outputs of all heads are spliced ​​together and the time information is fused through the fully connected layer; finally, the output is reconstructed to the original spatial dimension to obtain the temporal output features extracted by the temporal attention mechanism; the calculation process expression of the temporal output feature is: Calculate the association weights and perform weighted fusion: ， Fusion time information feature calculation: ， Time output feature calculation: ， in, is the attention feature output of the Nth head, () is the activation function, express The transpose of the matrix, They represent the batch size, channel dimension, time series length, feature height and width of the input features respectively, and D is the number of channels of each head. is the feature after integrating time information, For feature splicing processing, , is the learnable weight matrix, is the bias vector, is the feature reconstruction operation, Output features for time.

4. The method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention according to claim 1, characterized in that: The spatial attention module includes the following operation process: The feature data is subjected to spatial feature reconstruction processing; the two-dimensional spatial position code is decomposed into a horizontal direction code and a vertical direction code, and a learnable spatial position code is generated by adding outer products; the learnable spatial position code is added to the input tensor after the feature reconstruction processing, and the RGB and depth information features of the face with the spatial position code are output; the expression of the learnable spatial position code is: ， in, is a learnable spatial position encoding, For horizontal coding, For vertical coding, It indicates that its length is or A vector of all 1s, represents the Kronecker product; The facial RGB and depth information features with spatial position encoding are transformed into queries through three independent linear transformations. ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get , each head focuses on different spatial patterns; then the association weights in the spatial dimension are calculated independently for each head and weighted fusion is performed; the outputs of all heads are concatenated and the spatial information is fused through a fully connected layer; finally, the output is reconstructed to the original spatial dimension to obtain the spatial output features extracted by the spatial attention mechanism; the calculation process of the spatial output features is expressed as: Calculate the association weights and perform weighted fusion: ， Fusion spatial information feature calculation: ， Spatial output feature calculation: ， in, is the attention feature output of the Nth head, () is the activation function, express The transpose of the matrix, They represent the batch size, channel dimension, time series length, feature height and width of the input features respectively, and D is the number of channels of each head. , is the feature after integrating spatial information, For feature splicing processing, , is the learnable weight matrix, is the bias vector, S is the spatial feature dimension, is the spatial output feature, Reconstruct the operation for the feature.

5. The method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention according to claim 1, characterized in that: The channel attention module includes the following operation process: The feature data is orthogonally initialized with a 3D convolution kernel Filter out redundant channel information, compress along the spatiotemporal dimension, and extract the channel statistical feature tensor, which is expressed as: ， ， in, is the convolution kernel The filtered input tensor, X is the input tensor, In order to achieve the mapping from arbitrary size input to fixed size output by weighted summation and scale transformation of input features, Input feature tensor for the processed channels, Reconstruction operation for features; The channel statistical feature tensor is transformed into a query through three independent linear transformations ,key ,value ; will query ,key ,value Split along the channel dimension into Head, get , each head focuses on different channel modes; then the attention weight is calculated independently for each head; the attention weights of all heads are spliced ​​together and the channel information is fused through the fully connected layer to obtain the channel weight; finally, the channel weight is generated by the activation function. The channel mask is multiplied by the input tensor to obtain the channel output feature after the feature importance is distributed in the channel dimension; the calculation process expression of the channel output feature is: Calculate the association weights and perform weighted fusion: ， Fusion channel information feature calculation: ， Channel output feature calculation: ， in, is the attention feature output of the Nth head, () is the activation function, D is the number of channels of each head, , express The transpose of the matrix, Represent the batch size, channel dimension, time series length, feature height and width of the input features respectively, is the channel weight after fusing channel information, For feature splicing processing, , is the learnable weight matrix, is the bias vector, is the channel output feature, is the Sigmoid activation function.

6. The method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention according to claim 1, characterized in that: The multi-dimensional cross-gated fusion module includes the following operation process: The RGB features and depth information features of the face extracted by the time attention module, space attention module and channel attention module are respectively passed through the convolution layer to learn the weights of each dimension; the outputs of the three modules are superimposed after gated weighting, and are normalized through residual connection and layer normalization to obtain the three-dimensional fusion features of time, space and channel. ; The expression of the three-dimensional fusion feature of time, space and channel is: in, is the result of fusing the outputs of the three dimensions, θ represents the learnable parameters, and FusionFunc is a nonlinear mapping function. is the temporal output feature, is the spatial output feature, Output features for the channel.

7. The method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention according to claim 1, characterized in that: The TSC-CGAM model is trained using labeled face videos and depth information datasets, and after model convergence, a TSC-CGAM model capable of recognizing emotional anomalies is obtained.

8. The method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention according to claim 1, characterized in that: Also includes S4: S4: Extract one or more of the temporal output features, spatial output features, channel output features, and three-dimensional fusion features output by the TSC-CGA unit, convert them into a visual heat map, and superimpose the visual heat map on the original face RGB image to achieve feature map visualization.

9. The method for identifying abnormal emotion in 3D face videos based on multi-dimensional cross attention according to claim 8, characterized in that: The conversion of the visual heat map includes the following steps: Calculate the gradient weight of the feature to be converted; Calculate a 3D heat map of the feature to be converted according to the gradient weight; The 3D heat map is compressed into a 2D heat map and output as a visual heat map.

10. A device for identifying abnormal emotions in 3D face videos based on multi-dimensional cross-attention, characterized in that: It includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for identifying emotional anomalies in 3D facial videos based on multi-dimensional cross-attention according to any one of claims 1 to 9.

Images