Face video emotion evaluation method based on static emotion prior and space-time multi-view

Abstract

The invention discloses a facial video emotion evaluation method based on static emotion priori and spatio-temporal multi-view. Firstly, general facial features of a video to be evaluated and a static emotion image set are extracted and enhanced, and emotion features and state features are obtained. Emotional knowledge migration from the static atlas to the video is realized through the maximum mean value difference loss, and video global emotional features are obtained. Then, the long-range state features are divided into local areas, local state features are extracted through space attention, global-local interaction and local-local interaction are carried out through a multi-head attention mechanism, and frame-level state features are generated; and finally, obtaining a global variable based on the frame-level state features, and realizing emotion evaluation of the face video. According to the method, the multi-scale spatial-temporal features can be extracted robustly and efficiently, and the emotion evaluation precision is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a facial video emotion assessment method based on static emotion priors and spatiotemporal multi-view. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, automated sentiment analysis technology based on visual content has shown broad application prospects in various fields. Among them, facial expressions, as one of the most direct and richest natural signals conveying human emotional states, have become a core focus of sentiment assessment research.

[0003] Currently, methods for emotion assessment in facial videos primarily employ 3D convolutional neural networks, 2D convolutional neural networks combined with recurrent neural networks (RNNs) or long short-term memory architectures, as well as graph neural networks and visual Transformers. These methods aim to capture representations related to specific emotional states from dynamic facial changes. However, existing methods have the following key limitations: Existing publicly available facial video datasets generally lack frame-level emotion labels, making it difficult for models to fully learn the subtle temporal evolution of emotions and to adequately model the fine dynamic differences between different emotion categories. Furthermore, they only focus on overall facial dynamics, neglecting the correlation between fine local facial regions and multi-scale temporal sequences. Existing methods, such as the SPG framework, employ two-stage modeling to improve the ability to capture long-range spatiotemporal information, but their high computational complexity makes them unsuitable for real-time or resource-constrained applications. The MDDR method proposes pre-training on a static emotion dataset and then fine-tuning on a video depression dataset; however, while static emotion assessment and dynamic video emotion assessment are related tasks, their model-focused feature regions and spatiotemporal patterns differ, making direct fine-tuning prone to "catastrophic forgetting."

[0004] Furthermore, in real-world environments, significant changes in facial posture, partial facial occlusion (such as resting one's chin on one's hands), and uneven lighting are extremely common. Existing methods cannot effectively handle these situations, leading to unstable feature extraction and a significant decrease in evaluation performance.

[0005] In conclusion, current technology urgently needs a facial video emotion assessment method that can overcome the aforementioned limitations. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a facial video emotion assessment method based on static emotion prior and spatiotemporal multi-view to overcome or at least partially solve the above problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a facial video emotion assessment method based on static emotion priors and spatiotemporal multi-views, the steps of which include: Obtain facial videos and static emotion atlases to be evaluated, and extract common facial features from them respectively; Feature enhancement is performed on the respective common facial features to obtain the first emotion feature corresponding to the static emotion map, as well as the second emotion feature and long-range state feature corresponding to the facial video to be evaluated. Global emotion features are extracted from the first emotion feature and the second emotion feature respectively, and knowledge transfer is performed using the maximum mean difference loss to obtain the global emotion features of the video. The long-range state features are divided, and local state features of each local region are extracted based on spatial attention. Multi-head attention is used to perform global-local and local-local interactions between each local state feature and the global emotion features of the video to obtain frame-level state features. Based on the frame-level state features, global variables are obtained; based on the global variables, emotion assessment is performed on the facial video to be evaluated.

[0008] As a preferred embodiment, feature enhancement is performed on the respective common facial features, including the following steps: The general facial features of the facial video to be evaluated and the general facial features of the static emotion graph are mapped to the corresponding query vector, key vector and value vector, respectively. Calculate the dot product of the query vector and the corresponding key vector of the facial video to be evaluated to generate a video attention map; calculate the dot product of the query vector and the corresponding key vector of the static emotion graph to generate an atlas attention map. The video attention map and the atlas attention map are concatenated along the channel dimension to obtain a fused attention map; The first emotion feature is obtained by performing a dot product operation between the fused attention map and the value vector of the static emotion map set; the second emotion feature and long-range state feature are obtained by performing a dot product operation between the fused attention map and the value vector of the facial video to be evaluated.

[0009] Preferably, global emotion features are extracted from the first emotion feature and the second emotion feature, respectively, and the steps include: The first emotion feature is input into the first network branch to obtain the global emotion feature of the atlas; the second emotion feature is input into the second network branch to obtain the global emotion feature of the video. The first network branch is trained based on the first emotion feature, and the second network branch has the same structure as the first network branch and shares weights.

[0010] Preferably, the first network branch and / or the second network branch include: Initial attention weights are generated using the spatial attention module; The initial attention weights are simultaneously subjected to max pooling and average pooling, and after being concatenated and fused, a spatial attention weight map is generated through an activation function. The spatial attention weight map is multiplied by the initial attention weight and then added to the original input features. After global pooling, the global sentiment features corresponding to the original input features are obtained.

[0011] Preferably, global variables are obtained based on the frame-level state features, and the steps include: For the The frame-level state features of the layer are downsampled; The downsampled frame-level feature sequence is compared with the first... The global variables of the layer are concatenated and positional encoding is added to obtain the encoded features; The encoded features are subjected to temporal modeling processing to obtain temporal modeling output; The time-series modeling output is sequentially passed through a multilayer perceptron, a normalization layer, and a residual connection to obtain the first... Global variables of the layer and the first Frame-level state features of the layer; Repeat the above steps to obtain a global variable that integrates information from all time scales.

[0012] As a preferred approach, the facial video emotion assessment process is optimized by combining the regression loss of facial video emotion assessment, the classification cross-entropy loss of the first emotion feature, and the maximum mean difference loss of emotion knowledge transfer.

[0013] As a preferred option, the joint loss optimization function is expressed as:

[0014] In the formula, This indicates the regression loss in facial video emotion assessment. The classification cross-entropy loss represents the primary emotional feature. This represents the maximum mean difference loss in emotional knowledge transfer. This represents the weighting parameter.

[0015] Preferably, the regression loss for the facial video emotion assessment is:

[0016] In the formula, This indicates the number of facial video samples. Indicates the video sample index. This represents the true emotion assessment score of the i-th facial video sample. This represents the predicted emotion assessment score for the i-th facial video sample; The classification cross-entropy loss of the first emotion feature is:

[0017] In the formula, This indicates the number of samples in the static image set. Indicates the image sample index. Indicates the first The true emotion classification label of a static image sample Indicates the first Emotion classification prediction results for a static image sample; The maximum mean difference loss in the transfer of emotional knowledge is:

[0018] In the formula, Represents the global emotional features of an image. Indicates the overall emotional characteristics of the video. It is a kernel function.

[0019] Secondly, this embodiment provides a facial video emotion assessment system based on static emotion prior and spatiotemporal multiple views. This system applies the facial video emotion assessment method based on static emotion prior and spatiotemporal multiple views as described in any of the preceding embodiments, including: A general facial feature extraction module is used to acquire facial videos and static emotion atlases to be evaluated, and extract general facial features respectively. The task-aware fusion module is used to enhance the features of each general facial feature to obtain the first emotion feature corresponding to the static emotion map, as well as the second emotion feature and long-range state feature corresponding to the facial video to be evaluated. The static emotion knowledge transfer module is used to extract the global emotion features of the first emotion feature and the second emotion feature respectively, and to perform knowledge transfer using the maximum mean difference loss to obtain the global emotion features of the video. A multi-view spatial module is used to divide the long-range state features, extract local state features of each local region based on spatial attention, and use multi-head attention to perform global-local and local-local interactions between each local state feature and the global emotion features of the video to obtain frame-level state features. The emotion assessment module is used to obtain global variables based on the frame-level state features; and to perform emotion assessment on the facial video to be assessed based on the global variables.

[0020] This invention provides a facial video emotion assessment method based on static emotion priors and spatiotemporal multi-views. It aims to propose a novel model architecture and training strategy to extract multi-scale spatiotemporal features from facial videos in a more refined, robust, and efficient manner, thereby achieving high-precision emotion assessment.

[0021] Compared with the prior art, the beneficial effects of the above-mentioned technical solutions provided by the embodiments of the present invention include at least the following: 1. The feature enhancement process in this application, through feature decoupling and complementary information integration, enables the model to simultaneously consider task specificity and information sharing. The generated features not only meet the needs of each task, but are also rich in cross-task related information, providing more comprehensive feature support for accurate evaluation.

[0022] 2. To address the lack of frame-level basic emotion labels in facial video datasets, this invention proposes a cross-domain basic emotion knowledge transfer scheme: it solves the problem of effective utilization of unlabeled video data, achieves frame-level emotion feature extraction without additional manual annotation, significantly reduces the cost and threshold of technology implementation, and at the same time achieves accurate extraction of basic emotion features at the video frame level, providing core support for improving the performance of facial video evaluation.

[0023] 3. This application transforms frame-level features based on spatial attention and achieves inter-frame information interaction through multi-head attention, which enables dynamic fusion of global and local features, thereby effectively capturing the inconsistency of individual facial expressions and effective features in local occlusion scenarios, and improving feature robustness.

[0024] 4. By obtaining global variables through progressive temporal modeling, it can not only adaptively aggregate short-term emotion fluctuation features and long-term emotion stability features to capture the spatial distribution and temporal dynamic characteristics of facial emotions in all aspects, but also realize progressive temporal modeling from frame level to segment level by dynamically fusing temporal information at different time scales, providing comprehensive temporal feature support for the final facial video evaluation regression.

[0025] 5. This application uses joint loss for optimization, which enables end-to-end training, optimizes the facial video emotion recognition task, and improves the overall performance and generalization ability of the model. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1This is a flowchart of the facial video emotion assessment method based on static emotion prior and spatiotemporal multi-view provided in an embodiment of the present invention. Figure 2 This is a structural diagram of the task-aware fusion module provided in an embodiment of the present invention; Figure 3 This is a structural diagram of the static emotion knowledge transfer module provided in this embodiment of the invention; Figure 4 This is a structural diagram of the multi-view spatial module provided in the embodiments of the present invention; Figure 5 This is a structural diagram of the multi-scale timing module provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the use of an end-to-end joint loss function to optimize training, as provided in an embodiment of the present invention. Detailed Implementation The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] This invention discloses a facial video emotion assessment method based on static emotion priors and spatiotemporal multi-view, which can be used to assess various emotional states such as depression, melancholy, anxiety, stress, and fatigue. Figure 1 The steps include: Obtain facial videos and static emotion atlases to be evaluated, and extract common facial features from them respectively; Feature enhancement is performed on the respective common facial features to obtain the first emotion feature corresponding to the static emotion map, as well as the second emotion feature and long-range state feature corresponding to the facial video to be evaluated. Global emotion features are extracted from the first emotion feature and the second emotion feature respectively, and knowledge transfer is performed using the maximum mean difference loss to obtain the global emotion features of the video. The long-range state features are divided, and local state features of each local region are extracted based on spatial attention. Multi-head attention is used to perform global-local and local-local interactions between each local state feature and the global emotion features of the video to obtain frame-level state features. Based on the frame-level state features, global variables are obtained; based on the global variables, emotion assessment is performed on the facial video to be evaluated.

[0029] The following is a description through specific embodiments.

[0030] This embodiment uses a facial video dataset as the core evaluation data and a static facial emotion dataset as the source data for emotion knowledge transfer. First, the OpenFace open-source toolkit is used to preprocess both types of data: for the video dataset, the facial region of each frame is extracted to eliminate interference from complex backgrounds; for the static image dataset, the facial region is directly located. Then, all facial images are uniformly cropped and standardized to a size of 224×224 pixels to ensure consistent input data specifications.

[0031] To alleviate the overfitting problem caused by the small sample size, this embodiment performs data augmentation operations on video clip frames and static image samples. Specifically, this includes random horizontal flipping and random color jittering in four dimensions: brightness, contrast, saturation, and hue, to enhance the model's adaptability to changes in data distribution.

[0032] In one embodiment, preprocessed batch data (including 224×224×3×24 facial video data and 224×224×3 static emotional image data) are input into the ResNet backbone network, and general facial features of the two types of data are extracted respectively, including 24×24×128×24 video general facial features and 24×24×128 image general facial features.

[0033] In one optional embodiment, this application constructs a task-aware fusion module for feature enhancement of respective common facial features, the steps of which include: The general facial features of the facial video to be evaluated and the general facial features of the static emotion graph are mapped to the corresponding query vector, key vector and value vector, respectively. Calculate the dot product of the query vector and the corresponding key vector of the facial video to be evaluated to generate a video attention map; calculate the dot product of the query vector and the corresponding key vector of the static emotion graph to generate an atlas attention map. The video attention map and the atlas attention map are concatenated along the channel dimension to obtain a fused attention map; The first emotion feature is obtained by performing a dot product operation between the fused attention map and the value vector of the static emotion map set; the second emotion feature and long-range state feature are obtained by performing a dot product operation between the fused attention map and the value vector of the facial video to be evaluated.

[0034] This application decouples general facial features into task-specific features for facial video emotion assessment and task-specific features for image emotion recognition. This allows the model to retain the independent attributes of each task while efficiently integrating complementary information from the other task, thus improving the comprehensiveness of feature representation. This process can fully explore the inherent correlation between the two major tasks of image emotion recognition and facial video assessment.

[0035] In a preferred embodiment, the task-aware fusion module is structured as follows: Figure 2 As shown, to achieve task adaptation of features, general facial features are first passed through two parallel 3×3 convolutional layers to generate initial task features corresponding to "image emotion recognition" and "video emotion evaluation". Then, the two types of initial task features are mapped to query vector Q, key vector K and value vector V through 3×3 convolution respectively. Further, the Q and K of each task are used to perform dot product operation to generate special attention maps for the two types of tasks. After concatenating the two attention maps along the channel dimension, cross-task attention information sharing is achieved through 1×1 convolution to obtain a fused attention map. Finally, the fused attention map is multiplied with the V of the corresponding task to output a 24×24×128 first emotion feature (corresponding to the image emotion recognition task), a 24×24×128×24 second emotion feature, and a 24×24×128×24 long-range state feature (corresponding to the video evaluation task), which are both specific and complementary.

[0036] In an optional embodiment, to give the model accurate emotion feature extraction capabilities, global emotion features of the first emotion feature and the second emotion feature are extracted respectively, and knowledge transfer is performed using maximum mean difference loss to obtain global emotion features of the video.

[0037] In this embodiment, refer to Figure 3 A static emotion knowledge transfer module is set up, which includes a first network branch and a second network branch. The first network branch is trained based on the first emotion feature, and the second network branch has the same structure as the first network branch and shares weights.

[0038] Furthermore, this application inputs the first emotion feature into the first network branch to obtain a 512-dimensional global emotion feature of the graph; and inputs the second emotion feature into the second network branch to obtain a 512-dimensional global emotion feature of the video; at the same time, to address the problem of no frame-level emotion labels in the video dataset, the maximum mean difference (MMD) loss is used to achieve inter-domain knowledge transfer: that is, the global emotion feature of the graph and the global emotion feature of the video are input into the MMD loss function, and by minimizing the distribution difference of the two types of data in the feature space, the static graph emotion knowledge is transferred to the video task, so that the model can extract reliable emotion features from each frame of the video.

[0039] In a preferred embodiment of this invention, emotional features decoupled from general facial features are input into a spatial attention module for feature enhancement, highlighting key facial emotional regions; then, they are compressed into a fixed-dimensional emotional feature representation through a global pooling layer. Specifically, the extraction steps for both the first and second network branches include: Initial attention weights are generated using the spatial attention module; The initial attention weights are simultaneously subjected to max pooling and average pooling, and after being concatenated and fused, a spatial attention weight map is generated through an activation function. The spatial attention weight map is multiplied by the initial attention weight and then added to the original input features. After global pooling, the global sentiment features corresponding to the original input features are obtained.

[0040] In this embodiment, refer to Figure 3 The spatial attention module consists of two cascaded 3x3 convolutions. Optionally, before inputting into the spatial attention module, the input is first processed by two other cascaded 3x3 convolutions for preliminary feature extraction.

[0041] In an optional embodiment, the present invention designs a multi-view spatial module for dividing the long-range state features, extracting local state features of each local region based on spatial attention, and using multi-head attention to perform global-local and local-local interactions between each local state feature and the global emotion features of the video to obtain frame-level state features.

[0042] Specific steps are as follows: Figure 4 As shown, the long-range state features decoupled from the task-aware fusion module are divided into four non-overlapping parts. These parts are then transformed into 512-dimensional local state features using spatial attention and global pooling layers. Consequently, each frame of the video sample has four 512-dimensional state features and one 512-dimensional global video emotion feature. Furthermore, an attention module is used to perform global-local and local-local interactions, resulting in an enhanced 512-dimensional frame-level state feature for each frame of the video sample.

[0043] In an optional embodiment, the present invention further constructs a multi-scale temporal module for implementing progressive temporal modeling, the specific steps of which are described below. Figure 5 ,include: Let the total number of modules be n, and the first... During layer processing, the previous layer's frame-level state feature sequence is first processed by one-dimensional convolution. Downsampling: The formula is as follows:

[0044] downsampled frame count ; Then the downsampled frame-level feature sequence is compared with the first... The layer's global variables are concatenated, and positional encoding is added. This yields the encoded features; that is...

[0045] Furthermore, the encoded features are subjected to temporal modeling processing to obtain a temporal modeling output; and the temporal modeling output is sequentially passed through a multilayer perceptron, a normalization layer, and a residual connection to obtain the first... Global variables of the layer and the first Layer-level state features; the formula is as follows:

[0046] in To integrate the multi-scale temporal information into the l-th layer, after n layers of processing, the final global variables are... Integrate information from all timescales and use it as the core input for facial video emotion assessment.

[0047] To further optimize the above technical solution, this application combines the regression loss of facial video emotion assessment, the classification cross-entropy loss of the first emotion feature, and the maximum mean difference loss of emotion knowledge transfer to optimize the facial video emotion assessment process. The specific process is described in [reference needed]. Figure 6 , The joint loss is expressed as:

[0048] In the formula, This indicates the regression loss in facial video emotion assessment. The classification cross-entropy loss represents the primary emotional feature. This represents the maximum mean difference loss in emotional knowledge transfer. This represents the weighting parameter.

[0049] The regression loss for facial video emotion assessment is as follows:

[0050] In the formula, This indicates the number of facial video samples. Indicates the video sample index. This represents the true emotion assessment score of the i-th facial video sample. This represents the predicted emotion assessment score for the i-th facial video sample; The classification cross-entropy loss for the first emotion feature is generated by training on the static emotion features using emotion labels from the RAF-DB static dataset as supervision. The expression is as follows:

[0051] In the formula, This indicates the number of samples in the static image set. Indicates the image sample index. Indicates the first The true emotion classification label of a static image sample Indicates the first The emotion classification prediction results of a static image sample. In this embodiment, the prediction of seven basic emotions includes happiness, anger, fear, sadness, surprise, disgust and contempt. The maximum mean difference loss for emotional knowledge transfer is:

[0052] In the formula, Represents the global emotional features of an image. Indicates the overall emotional characteristics of the video. It is a kernel function.

[0053] In an embodiment of the present invention, the parameters for model training are configured as follows: the batch size is set to 16, and the initial learning rate is 1×10⁻⁶. - 3. The AdamW optimization algorithm (Adam WeightDecay) and a cosine learning rate scheduler were selected. In the initial training phase, specifically the first 10 epochs, a cosine annealing strategy was used for warm-up to gradually adjust the learning rate, thereby improving the model's convergence stability. On an RTX 3090 GPU, the total number of training epochs was set between 100 and 200, and an early stopping strategy was implemented.

[0054] Based on the same inventive concept, embodiments of the present invention also provide a facial video emotion assessment system based on static emotion prior and spatiotemporal multiple views. This system applies the facial video emotion assessment method based on static emotion prior and spatiotemporal multiple views as described in any of the preceding claims, including: A general facial feature extraction module is used to acquire facial videos and static emotion atlases to be evaluated, and extract general facial features respectively. The task-aware fusion module is used to enhance the features of each general facial feature to obtain the first emotion feature corresponding to the static emotion map, as well as the second emotion feature and long-range state feature corresponding to the facial video to be evaluated. The static emotion knowledge transfer module is used to extract the global emotion features of the first emotion feature and the second emotion feature respectively, and to perform knowledge transfer using the maximum mean difference loss to obtain the global emotion features of the video. A multi-view spatial module is used to divide the long-range state features, extract local state features of each local region based on spatial attention, and use multi-head attention to perform global-local and local-local interactions between each local state feature and the global emotion features of the video to obtain frame-level state features. The emotion assessment module is used to obtain global variables based on the frame-level state features; and to perform emotion assessment on the facial video to be assessed based on the global variables.

[0055] In this embodiment, the execution steps of each module are consistent with the steps in the aforementioned facial video emotion assessment method based on static emotion prior and spatiotemporal multi-view. Therefore, detailed steps can be found in the implementation of the aforementioned method, and repeated parts will not be described again.

[0056] This invention provides a facial video emotion assessment method based on static emotion priors and spatiotemporal multi-views. It can be executed by a computer program and mainly proposes a novel model architecture and training strategy to extract multi-scale spatiotemporal features from facial videos in a more refined, robust and efficient manner, thereby achieving high-precision facial video emotion assessment.

[0057] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0058] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A facial video emotion assessment method based on static emotion priors and spatiotemporal multi-views, characterized in that, include: Obtain facial videos and static emotion atlases to be evaluated, and extract common facial features from them respectively; Feature enhancement is performed on the respective common facial features to obtain the first emotion feature corresponding to the static emotion map, as well as the second emotion feature and long-range state feature corresponding to the facial video to be evaluated. Global emotion features are extracted from the first emotion feature and the second emotion feature respectively, and knowledge transfer is performed using the maximum mean difference loss to obtain the global emotion features of the video. The long-range state features are divided, and local state features of each local region are extracted based on spatial attention. Multi-head attention is used to perform global-local and local-local interactions between each local state feature and the global emotion features of the video to obtain frame-level state features. Based on the frame-level state features, global variables are obtained; based on the global variables, emotion assessment is performed on the facial video to be evaluated.

2. The facial video emotion assessment method according to claim 1, characterized in that, Feature enhancement is performed on each common facial feature, and the steps include: The general facial features of the facial video to be evaluated and the general facial features of the static emotion graph are mapped to the corresponding query vector, key vector and value vector, respectively. Calculate the dot product of the query vector and the corresponding key vector of the facial video to be evaluated to generate a video attention map; calculate the dot product of the query vector and the corresponding key vector of the static emotion graph to generate an atlas attention map. The video attention map and the atlas attention map are concatenated along the channel dimension to obtain a fused attention map; The first emotion feature is obtained by performing a dot product operation between the fused attention map and the value vector of the static emotion map set; the second emotion feature and long-range state feature are obtained by performing a dot product operation between the fused attention map and the value vector of the facial video to be evaluated.

3. The facial video emotion assessment method according to claim 1, characterized in that, Extracting global emotion features from the first emotion feature and the second emotion feature respectively, the steps include: The first emotion feature is input into the first network branch to obtain the global emotion feature of the atlas; the second emotion feature is input into the second network branch to obtain the global emotion feature of the video. The first network branch is trained based on the first emotion feature, and the second network branch has the same structure as the first network branch and shares weights.

4. The facial video emotion assessment method according to claim 3, characterized in that, The first network branch and / or the second network branch include: Initial attention weights are generated using the spatial attention module; The initial attention weights are simultaneously subjected to max pooling and average pooling, and after being concatenated and fused, a spatial attention weight map is generated through an activation function. The spatial attention weight map is multiplied by the initial attention weight and then added to the original input features. After global pooling, the global sentiment features corresponding to the original input features are obtained.

5. The facial video emotion assessment method according to claim 1, characterized in that, Based on the frame-level state features, global variables are obtained, including the following steps: For the The frame-level state features of the layer are downsampled; The downsampled frame-level feature sequence is compared with the first... The global variables of the layer are concatenated and positional encoding is added to obtain the encoded features; The encoded features are subjected to temporal modeling processing to obtain temporal modeling output; The time-series modeling output is sequentially passed through a multilayer perceptron, a normalization layer, and a residual connection to obtain the first... Global variables of the layer and the first Frame-level state features of the layer; Repeat the above steps to obtain a global variable that integrates information from all time scales.

6. The facial video emotion assessment method according to claim 1, characterized in that, The facial video emotion assessment process is optimized by combining regression loss from facial video emotion assessment, classification cross-entropy loss from the first emotion feature, and maximum mean difference loss from emotion knowledge transfer.

7. The facial video emotion assessment method according to claim 6, characterized in that, The joint loss optimization function is expressed as: In the formula, This indicates the regression loss in facial video emotion assessment. The classification cross-entropy loss represents the primary emotional feature. This represents the maximum mean difference loss in emotional knowledge transfer. This represents the weighting parameter.

8. The facial video emotion assessment method according to claim 6, characterized in that, The regression loss for the facial video emotion assessment is: In the formula, This indicates the number of facial video samples. Indicates the video sample index. This represents the true emotion assessment score of the i-th facial video sample. This represents the predicted emotion assessment score for the i-th facial video sample; The classification cross-entropy loss of the first emotion feature is: In the formula, This indicates the number of samples in the static image set. Indicates the image sample index. Indicates the first The true emotion classification label of a static image sample Indicates the first Emotion classification prediction results for a static image sample; The maximum mean difference loss in the transfer of emotional knowledge is: In the formula, Represents the global emotional features of an image. Indicates the overall emotional characteristics of the video. It is a kernel function.

9. A facial video emotion assessment system based on static emotion priors and spatiotemporal multi-view, characterized in that, The facial video emotion assessment method based on static emotion prior and spatiotemporal multi-view as described in any one of claims 1-8 includes: A general facial feature extraction module is used to acquire facial videos and static emotion atlases to be evaluated, and extract general facial features respectively. The task-aware fusion module is used to enhance the features of each general facial feature to obtain the first emotion feature corresponding to the static emotion map, as well as the second emotion feature and long-range state feature corresponding to the facial video to be evaluated. The static emotion knowledge transfer module is used to extract the global emotion features of the first emotion feature and the second emotion feature respectively, and to perform knowledge transfer using the maximum mean difference loss to obtain the global emotion features of the video. A multi-view spatial module is used to divide the long-range state features, extract local state features of each local region based on spatial attention, and use multi-head attention to perform global-local and local-local interactions between each local state feature and the global emotion features of the video to obtain frame-level state features. The emotion assessment module is used to obtain global variables based on the frame-level state features; and to perform emotion assessment on the facial video to be assessed based on the global variables.

Images