Emotion detection method and apparatus

Abstract

The present disclosure relates to an emotion detection method and device. The method comprises: performing standardization processing on an input video frame by frame to obtain a plurality of single-frame face images; processing the single-frame face images to extract frame-level features of the single-frame face images; identifying a time point of emotion transition in the output video, and segmenting the output video based on the time point to obtain a plurality of video sub-clips; for each video sub-clip, aggregating the frame-level features of the single-frame face images within the corresponding video sub-clip through a time sequence dynamic adapter to generate aggregated video features; inputting the aggregated video features of the corresponding video sub-clip into a large language model to generate a text describing an emotional state; and analyzing the text using a text classification regression model to output an emotion category and a corresponding emotion intensity value of each video sub-clip. Thus, the dynamic change process of facial emotions in the video can be accurately captured and analyzed, and the computational efficiency is guaranteed and the result interpretability is improved.

Description

Technical Field

[0001] This disclosure relates to the field of dynamic facial expression recognition technology, and more specifically, to an emotion detection method and apparatus. Background Technology

[0002] Dynamic facial expression recognition aims to understand human emotions by analyzing facial changes in video sequences and is one of the key technologies for human-computer interaction. Traditional methods typically use convolutional neural networks (CNNs) to extract features and combine them with recurrent neural networks (RNNs) or 3D convolutional networks (3D convolutional networks) to model temporal information. However, the design of CNNs and RNNs often performs poorly when capturing complex long-term and short-term dependencies, while 3D convolutional networks face huge computational overhead and a large number of parameters, making them difficult to deploy on resource-constrained devices.

[0003] In recent years, visual-language pre-trained models, represented by CLIP (Contrastive Language-Image Pretraining), have demonstrated great potential in various visual tasks due to their powerful zero-shot generalization ability and have begun to be applied to the field of dynamic facial expression recognition. Existing methods attempt to adapt CLIP models by introducing temporal modules or designing specialized text prompts. Despite some progress, these methods still have significant drawbacks: First, to adapt to downstream tasks, it is usually necessary to completely fine-tune the large visual encoder, resulting in extremely high training costs, which defeats the purpose of pre-trained models; second, the temporal modeling methods used (e.g., average pooling or single-layer Transformer in the time dimension) are relatively simple and cannot effectively capture the subtle and non-linear dynamics in the process of facial expression changes; third, the output dimension of existing methods is relatively singular, mostly only able to provide an overall emotion classification label for the entire video segment, unable to identify and output possible emotion transitions within the same video segment, nor can they quantify changes in the intensity of emotion. Summary of the Invention

[0004] To address the aforementioned issues, this disclosure proposes an emotion detection method and apparatus, a computing system, and a computer-readable storage medium.

[0005] According to one aspect of this disclosure, an emotion detection method is provided, the emotion detection method comprising: performing frame-by-frame standardization processing on an input video to obtain multiple single-frame face images as an output video; processing the single-frame face images using a visual encoder to extract frame-level features of the single-frame face images; identifying the time points of emotion transitions in the output video, and segmenting the output video based on the time points to obtain multiple video sub-segments; for each video sub-segment, aggregating the frame-level features of the single-frame face images within the corresponding video sub-segment using a temporal dynamic adapter to generate aggregated video features of the corresponding video sub-segment; inputting the aggregated video features of the corresponding video sub-segment into a large language model to generate text describing the emotional state; and analyzing the text using a text classification and regression model to output the emotion category and corresponding emotion intensity value of each video sub-segment.

[0006] Optionally, the input video is a video clip containing a face or a real-time video stream.

[0007] Optionally, the standardization process includes face detection, feature point extraction, face alignment, and cropping operations.

[0008] Optionally, the step of processing the single-frame face image using a visual encoder to extract frame-level features of the single-frame face image includes: uniformly dividing the single-frame face image into multiple non-overlapping image blocks; flattening and linearly projecting the multiple non-overlapping image blocks to generate an initial feature sequence; and generating frame-level features of the single-frame face image based on the initial feature sequence using a transformer, wherein the transformer includes one or more transformer layers connected in sequence, each transformer layer including a multi-head self-attention layer and a lightweight shared adapter connected to the multi-head self-attention layer.

[0009] Optionally, the step of identifying the time point of emotion transition in the output video includes: determining the central feature vector of the feature vectors of the frame-level features of multiple single-frame face images included in the sliding time window; determining the feature variation dispersion score of the feature vectors of the frame-level features of the multiple single-frame face images, wherein the feature variation dispersion score is the average cosine distance from the feature vector of the corresponding single-frame face image to the central feature vector; and determining the start time of the continuous sequence as the time point when the single-frame face images corresponding to the feature vectors with feature variation dispersion scores greater than a predetermined threshold in the sliding time window constitute a continuous sequence and the number of single-frame face image frames included in the continuous sequence is greater than or equal to a predetermined number.

[0010] Optionally, the temporal dynamic adapter includes a gated recurrent unit and a dynamic scaler. The dynamic scaler is configured to: for each single-frame face image, generate a corresponding scaling factor based on the frame-level features processed by the gated recurrent unit through a learnable linear transformation layer; multiply the generated scaling factor element-wise with the processed frame-level features to obtain the weighted features of the corresponding single-frame face image; and aggregate the weighted frame-level features of the single-frame face images within the corresponding video sub-segment to generate the aggregated video features of the corresponding video sub-segment.

[0011] Optionally, the step of analyzing the text using a text classification and regression model to output the emotion category and corresponding emotion intensity value of each video segment includes: mapping the text sequence into sentence-level feature vectors; inputting the sentence-level feature vectors into a classification head and a regression head in parallel to output the emotion category and corresponding emotion intensity value of each video segment, wherein the classification head includes a fully connected layer and a Softmax activation function to transform the sentence-level feature vectors into a probability distribution corresponding to a predefined emotion and taking the emotion corresponding to the highest probability as the emotion category of the corresponding video segment, and wherein the regression head includes a fully connected layer and a Sigmoid activation function to transform the sentence-level feature vectors into continuous values ​​within a predetermined range as the emotion intensity value of the corresponding video segment.

[0012] According to another aspect of this disclosure, an emotion detection device is provided, comprising: a video preprocessing unit configured to perform frame-by-frame normalization processing on an input video to obtain multiple single-frame face images as output videos; a dynamic temporal feature extraction and aggregation unit configured to process the single-frame face images through a visual encoder to extract frame-level features of the single-frame face images, identify the time points of emotion transitions in the output video, and segment the output video based on the time points to obtain multiple video sub-segments, and for each video sub-segment, aggregate the frame-level features of the single-frame face images within the corresponding video sub-segment through a temporal dynamic adapter to generate aggregated video features of the corresponding video sub-segment; and an emotion description generation and classification unit configured to input the aggregated video features of the corresponding video sub-segments into a large language model to generate text describing the emotional state, analyze the text using a text classification regression model, and output the emotion category and corresponding emotion intensity value of each video sub-segment.

[0013] Optionally, the input video is a video clip containing a face or a real-time video stream.

[0014] Optionally, the standardization process includes face detection, feature point extraction, face alignment, and cropping operations.

[0015] Optionally, the dynamic temporal feature extraction and aggregation unit is configured to: uniformly segment the single-frame face image into multiple non-overlapping image blocks; flatten and linearly project the multiple non-overlapping image blocks to generate an initial feature sequence; and generate frame-level features of the single-frame face image based on the initial feature sequence using a transformer, wherein the transformer includes one or more transformer layers connected in sequence, each transformer layer including a multi-head self-attention layer and a lightweight shared adapter connected to the multi-head self-attention layer.

[0016] Optionally, the dynamic temporal feature extraction and aggregation unit is configured to: determine the central feature vector of the feature vectors of the frame-level features of multiple single-frame face images included in the sliding time window; determine the feature variation dispersion score of the feature vectors of the frame-level features of the multiple single-frame face images, wherein the feature variation dispersion score is the average cosine distance from the feature vector of the corresponding single-frame face image to the central feature vector; and, in response to the single-frame face images corresponding to the feature vectors whose feature variation dispersion scores are greater than a predetermined threshold in the sliding time window constitute a continuous sequence and the number of single-frame face image frames included in the continuous sequence is greater than or equal to a predetermined number, determine the start time of the continuous sequence as the time point.

[0017] Optionally, the temporal dynamic adapter includes a gated recurrent unit and a dynamic scaler. The dynamic scaler is configured to: for each single-frame face image, generate a corresponding scaling factor based on the frame-level features processed by the gated recurrent unit through a learnable linear transformation layer; multiply the generated scaling factor element-wise with the processed frame-level features to obtain the weighted features of the corresponding single-frame face image; and aggregate the weighted frame-level features of the single-frame face images within the corresponding video sub-segment to generate the aggregated video features of the corresponding video sub-segment.

[0018] Optionally, the emotion description generation and classification unit is configured to: map the sequence of the text into sentence-level feature vectors; input the sentence-level feature vectors in parallel into a classification head and a regression head to output the emotion category and corresponding emotion intensity value of each video sub-segment, wherein the classification head includes a fully connected layer and a Softmax activation function, used to transform the sentence-level feature vectors into a probability distribution corresponding to a predefined emotion and take the emotion corresponding to the one with the highest probability as the emotion category of the corresponding video sub-segment, and wherein the regression head includes a fully connected layer and a Sigmoid activation function, used to transform the sentence-level feature vectors into continuous values ​​within a predetermined range, as the emotion intensity value of the corresponding video sub-segment.

[0019] According to another aspect of this disclosure, a computing system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the emotion detection method as described above.

[0020] According to another aspect of this disclosure, a computer-readable storage medium for storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device causes the at least one computing device to perform the emotion detection method as described above.

[0021] By employing this disclosure, it is possible to accurately capture and analyze the dynamic changes in facial emotions in videos (including, for example, the type, intensity, and timing of the emotion), while ensuring computational efficiency and improving the interpretability of the results. Attached Figure Description

[0022] The above and / or other objects and advantages of this disclosure will become clearer from the following description of embodiments in conjunction with the accompanying drawings, wherein:

[0023] Figure 1 This is a flowchart illustrating an exemplary embodiment of an emotion detection method according to the present disclosure;

[0024] Figure 2 This is a flowchart illustrating video preprocessing steps according to an embodiment of the present disclosure;

[0025] Figure 3 This is a flowchart illustrating the dynamic temporal feature extraction and aggregation steps according to an embodiment of the present disclosure;

[0026] Figure 4 This is a flowchart illustrating the steps of generating and classifying emotion descriptions according to embodiments of the present disclosure;

[0027] Figure 5 This is a block diagram illustrating an emotion detection device according to an exemplary embodiment of the present disclosure;

[0028] Figure 6 This is a block diagram illustrating a computing system including at least one computing device and at least one storage device of storage instructions according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0029] The following description, in conjunction with the accompanying drawings, provides specific embodiments to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding this disclosure, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be altered as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0030] This disclosure addresses the inaccuracies of existing methods in capturing dynamic changes in emotions and the limited dimensionality of their outputs. It proposes a dynamic feature aggregation analysis method based on emotion change point perception, which significantly improves performance compared to existing deep learning models. The proposed emotion detection framework based on dynamic temporal aggregation includes a video preprocessing module, a dynamic temporal feature extraction and aggregation module, and an emotion description generation and classification module. It can be widely applied to scenarios requiring real-time and accurate understanding of human emotional states, such as human-computer interaction, smart cockpits, mental health assessment, distance education, and market analysis. The video preprocessing module performs standardized operations such as face detection, keypoint extraction, face alignment, and image cropping on each frame of the input video stream or video segment. The dynamic temporal feature extraction and aggregation module's processing can be divided into three stages: frame-level feature extraction, emotion change point detection, and segmented temporal feature aggregation. The dynamic temporal feature extraction and aggregation module first extracts features from frame-by-frame images using a visual encoder integrated with a lightweight adapter. Then, it calculates the similarity of feature distributions within adjacent time windows to detect key time points of emotional change. Finally, it segments the video based on these key time points and aggregates the multi-frame features within each segment using a temporal adapter with a dynamic scaling mechanism, thus effectively segmenting the video to capture emotions. The emotion description generation and classification module uses a large language model to convert the aggregated video features into natural language text. It then analyzes the text using a dual-output text classification and regression model, ultimately outputting the emotion category and intensity.

[0031] Figure 1 This is a flowchart illustrating an emotion detection method according to an exemplary embodiment of the present disclosure. Figure 2 This is a flowchart illustrating video preprocessing steps according to an embodiment of the present disclosure. Figure 3 This is a flowchart illustrating the dynamic temporal feature extraction and aggregation steps according to an embodiment of the present disclosure. Figure 4 This is a flowchart illustrating the steps of generating and classifying emotion descriptions according to embodiments of the present disclosure.

[0032] like Figure 1As shown, in step S101, the input video is standardized frame by frame to obtain multiple single-frame face images as the output video. In this example, the input video can be a video clip containing faces or a real-time video stream. In this example, the standardization process includes face detection, feature point extraction, face alignment, and cropping operations.

[0033] Step S101 is used to standardize the input video source to prepare for subsequent feature extraction. An example of its specific process is shown below. Figure 2 As shown. In step 201, face detection is performed. For example, each frame of the input video is processed, and a face detection algorithm is used to locate the bounding box of the face region. If a face is detected in the current frame in step S202, proceed to step S203; otherwise, skip the current frame. In step S203, feature point extraction is performed. For example, within the detected face region, five key facial feature points (including the corners of the eyes, the tip of the nose, and the corners of the mouth) are further located. In step S204, face alignment and cropping are performed. For example, based on the positions of the key points determined in step S203, the face is corrected using an affine transformation; the aligned face region is cropped from the original image and uniformly scaled to a specific size (e.g., 224x224 pixels) as input for subsequent processing.

[0034] In step S102, the single-frame face image is processed by a visual encoder to extract frame-level features. In step S103, the time points of emotional transitions in the output video are identified, and the output video is segmented based on these time points to obtain multiple video sub-segments. In step S104, for each video sub-segment, the frame-level features of the single-frame face image within the corresponding video sub-segment are aggregated using a temporal dynamic adapter to generate aggregated video features for the corresponding video sub-segment. For example, firstly, a visual encoder containing a lightweight shared adapter (e.g., a vision model based on the Transformer architecture (ViT)) is used to process the single-frame face image to extract frame-level features for each frame; then, a sliding window algorithm is used to identify key time points of emotional transitions, and the video is segmented based on the identified time points; finally, a temporal dynamic adapter containing a gated loop unit and a dynamic scaling mechanism is used to aggregate the frame-level features within the sub-segments to generate aggregated video features that represent the video sub-segment. Steps S102 to S104 are used to extract deep features that can characterize dynamic emotions from the preprocessed video frame sequence. A specific example of the process is as follows: Figure 3 As shown.

[0035] In the example, a single-frame face image is uniformly divided into multiple non-overlapping image patches; these non-overlapping image patches are flattened and linearly projected to generate an initial feature sequence; and a transformer is used to generate frame-level features of the single-frame face image based on the initial feature sequence. The transformer includes one or more sequentially connected transformer layers, each transformer layer including a multi-head self-attention layer and a lightweight shared adapter connected to the multi-head self-attention layer. For example, Figure 3 The frame-level feature extraction stage in step S101 is used to extract frame-level features from each frame of standardized face image output by a visual encoder. The processing flow may include: performing image segmentation on a single frame of face image, and flattening and linearly projecting the image blocks to generate an initial feature sequence; inputting the initial feature sequence into an encoder network consisting of cascaded transformer layers, where each transformer layer consists of a multi-head self-attention module and a lightweight shared adapter module, and uses residual connections; finally, the visual encoder outputs a vector for each frame as the feature of that frame.

[0036] For example, a visual encoder used to extract frame-level features applies this to each normalized single-frame face image. Processing is performed to generate frame-level features. The processing includes: processing single-frame face images Divide evenly into The size is Non-overlapping image patches, in which Then, the M image patches are flattened and linearly projected to generate a dimension of initial feature sequence (For example, for these M image patches, each patch is flattened and then input into a linear layer. The linear layer maps the flattened patch into a d-dimensional feature space, where d is the dimension of the linear layer's output.) Transformer processing consists of one or more transformer layers connected in sequence. Each transformer layer consists of a multi-head self-attention layer and a lightweight shared adapter connected thereafter.

[0037] For example, the transformer will initialize the feature sequence. Enter in sequence The process is performed using a series of converter layers, where the first... layer Its input features Transform into output features The transformation process consists of the following steps:

[0038] First, the multi-head self-attention layer applies input features. Calculate intermediate features :

[0039]

[0040] Among them, each head Defined as:

[0041]

[0042] In the formula, , , The first The query, key, and value projection matrix of the size. For the number of heads, This is the output projection matrix of the multi-head self-attention layer.

[0043] Secondly, intermediate features are accessed through a lightweight shared adapter. Optimization is performed to obtain the adapter output characteristics. :

[0044]

[0045] in, For a dimension-reduced linear projection matrix, For an upgraded linear projection matrix, It is a non-linear activation function.

[0046] Finally, the adapter output characteristics With the Layer input features The final output characteristics of this converter layer are obtained by adding the residual connections. :

[0047]

[0048] Figure 3 The emotion change point detection stage processes the frame-level feature sequence output from the frame-level feature extraction stage to identify the time points of emotion change. In the example, the central feature vector of the feature vectors of the frame-level features of multiple single-frame face images included within the sliding time window is determined; the feature change dispersion score of the feature vectors of the frame-level features of the multiple single-frame face images is determined, where the feature change dispersion score is the mean cosine distance from the feature vector of the corresponding single-frame face image to the central feature vector; in response to the single-frame face images corresponding to feature vectors whose feature change dispersion scores within the sliding time window are greater than a predetermined threshold forming a continuous sequence and the number of single-frame face images included in the continuous sequence being greater than or equal to a predetermined number, the start time of the continuous sequence is determined as the time point of emotion change.

[0049] For example, firstly, for a size of Sliding time window Calculate the center eigenvector of all frame-level eigenvectors within this window. :

[0050]

[0051] Secondly, calculate the feature variation dispersion score within the window. It is defined as the feature vector of all frames within the window. To the central feature vector Mean cosine distance:

[0052]

[0053] Set a predetermined threshold for change If and only if there are consecutive [windows] within the window. Frame change score satisfies At that time, it will be composed of continuous The starting point of a continuous sequence of frames The point is marked as the emotional change point (i.e., the time point of emotional shift).

[0054] In the example, the temporal dynamic adapter includes a gated recurrent unit and a dynamic scaler. The dynamic scaler is configured to: for each single-frame face image, generate a corresponding scaling factor based on the frame-level features processed by the gated recurrent unit through a learnable linear transform layer; multiply the generated scaling factor element-wise with the processed frame-level features to obtain the weighted features of the corresponding single-frame face image; and aggregate the weighted frame-level features of the single-frame face images within the corresponding video sub-segment to generate the aggregated video features of the corresponding video sub-segment. For example, Figure 3 The segmented temporal feature set stage in the video is used to detect the time points of emotional changes. It divides the entire video's frame feature sequence into several sub-segments and aggregates features for each segment using a temporal dynamic adapter. The temporal dynamic adapter can consist of a gated recurrent unit (GRU) and a dynamic scaler. After the GRU models the temporal information, the dynamic scaler calculates the weights and performs weighted aggregation to generate a single aggregated video feature. Thus, for each video sub-segment, an aggregated feature is obtained.

[0055] For example, a temporal dynamic adapter for feature aggregation includes a gated loop unit and a dynamic scaler. The gated loop unit captures the temporal dependencies between features of each frame within a video segment, and the dynamic scaler adaptively adjusts the contribution weight of each frame's features in the aggregation process. Specifically, let the features of each frame after processing by the gated loop unit be... The dynamic scaling mechanism achieves adaptive weighting of the feature through the following formulaic steps.

[0056] First, a learnable linear transformation layer is used to target the features. Generate a corresponding nonnegative scaling factor Its calculation method is defined by the following formula:

[0057]

[0058] in, It is a non-negative activation function to ensure that the value of the scaling factor is non-negative.

[0059] Secondly, the generated scaling factor With features Perform element-wise multiplication to obtain the final weighted features of the frame. This allows for adaptive adjustment of feature weights for different frames, highlighting frames with richer information and suppressing the influence of irrelevant variables.

[0060]

[0061] Here, ⊙ represents element-wise multiplication (Hadamard product).

[0062] In step S105, the aggregated video features of the corresponding video segments are input into a large language model to generate text describing the emotional state. In step S106, a text classification and regression model is used to analyze the text to output the emotion category and corresponding emotion intensity value for each video segment.

[0063] In the example, the text sequence can be mapped to sentence-level feature vectors. These sentence-level feature vectors are then input in parallel into a classification head and a regression head to output the emotion category and corresponding emotion intensity value for each video segment. The classification head includes a fully connected layer and a Softmax activation function to transform the sentence-level feature vectors into a probability distribution corresponding to a predefined emotion, and the emotion corresponding to the highest probability is taken as the emotion category of the corresponding video segment. The regression head includes a fully connected layer and a Sigmoid activation function to transform the sentence-level feature vectors into continuous values ​​within a predetermined range, which are then used as the emotion intensity value of the corresponding video segment. For example, as shown... Figure 4As shown, in step S401, emotion description generation can be performed. Specifically, aggregated video features and a preset text prompt (a fixed text that provides contextual information to guide the large language model to perform a specific text generation task, such as "Please describe this person's current emotional state in one sentence") can be input into a pre-trained large language model (a pre-trained generative model used to decode non-linguistic aggregated video features into natural language text sequences). The large language model then generates natural language text describing the emotional state. In step S402, text classification and regression are performed. Specifically, the natural language text generated in step S401 is input into a text analysis model with dual output heads. The classification head of the text analysis model outputs one of several predefined emotion categories, and the regression head of the text analysis model outputs an emotion intensity value within a predetermined range (e.g., between 0 and 1).

[0064] For example, firstly, for each video segment, the aggregated video features are input into a pre-trained large language model, and combined with pre-defined prompts, a natural language text sentence describing the emotional state of the characters in the video is generated; secondly, the natural language text is input into a BERT (Bidirectional Encoder Representations)-based model. The text analysis model, fine-tuned using the Transformers architecture, processes the input natural language text as follows: First, the model uses its internal multi-layer Transformer encoder to process the text sequence, mapping it into a fixed-dimensional sentence-level feature vector that comprehensively captures its semantic information. Then, the sentence-level feature vector is input in parallel into two independent output heads for final sentiment analysis. The classification head consists of a fully connected layer and a Softmax activation function, transforming the sentence-level feature vector into a seven-dimensional probability distribution. This distribution corresponds to the probabilities of seven predefined emotions (anger, disgust, fear, happiness, neutrality, sadness, and surprise), and the one with the highest probability is taken as the final emotion category. The regression head consists of a fully connected layer and a Sigmoid activation function, transforming the sentence-level feature vector into a continuous value between 0 and 1, which serves as the final sentiment intensity value (i.e., the sentiment intensity value of the corresponding video segment).

[0065] By employing the emotion detection method according to exemplary embodiments of the present disclosure, it is possible to effectively adapt pre-trained visual-language models without sacrificing performance, enhance their temporal modeling capabilities specifically for dynamic expression recognition tasks, and provide multi-dimensional and highly interpretable outputs.

[0066] Figure 5 This is a block diagram illustrating an emotion detection device according to an exemplary embodiment of the present disclosure.

[0067] like Figure 5 As shown, the emotion detection apparatus 500 according to an exemplary embodiment of the present disclosure includes: a video preprocessing unit 501, configured to perform frame-by-frame normalization processing on the input video to obtain multiple single-frame face images as output videos; a dynamic temporal feature extraction and aggregation unit 502, configured to process the single-frame face images through a visual encoder to extract frame-level features of the single-frame face images, identify the time points of emotion transitions in the output video, and segment the output video based on the time points to obtain multiple video sub-segments, and for each video sub-segment, aggregate the frame-level features of the single-frame face images in the corresponding video sub-segment through a temporal dynamic adapter to generate aggregated video features of the corresponding video sub-segment; and an emotion description generation and classification unit 503, configured to input the aggregated video features of the corresponding video sub-segment into a large language model to generate text describing the emotional state, analyze the text using a text classification regression model, and output the emotion category and corresponding emotion intensity value of each video sub-segment.

[0068] In the example, the input video can be a video clip containing a face or a live video stream.

[0069] In the example, standardization processes may include face detection, feature point extraction, face alignment, and cropping operations.

[0070] In the example, the dynamic temporal feature extraction and aggregation unit 502 can be configured to: uniformly segment a single-frame face image into multiple non-overlapping image blocks; flatten and linearly project the multiple non-overlapping image blocks to generate an initial feature sequence; and generate frame-level features of the single-frame face image based on the initial feature sequence through a transformer, wherein the transformer includes one or more transformer layers connected in sequence, each transformer layer including a multi-head self-attention layer and a lightweight shared adapter connected to the multi-head self-attention layer.

[0071] In the example, the dynamic temporal feature extraction and aggregation unit 502 can be configured to: determine the central feature vector of the feature vectors of the frame-level features of multiple single-frame face images included in the sliding time window; determine the feature variation dispersion score of the feature vectors of the frame-level features of multiple single-frame face images, wherein the feature variation dispersion score is the average cosine distance from the feature vector of the corresponding single-frame face image to the central feature vector; and, in response to the single-frame face images corresponding to the feature vectors whose feature variation dispersion scores are greater than a predetermined threshold in the sliding time window constitute a continuous sequence and the number of single-frame face image frames included in the continuous sequence is greater than or equal to a predetermined number, determine the start time of the continuous sequence as the time point.

[0072] In the example, the temporal dynamic adapter includes a gated recurrent unit and a dynamic scaler. The dynamic scaler is configured to: for each single-frame face image, generate a corresponding scaling factor based on the frame-level features processed by the gated recurrent unit through a learnable linear transformation layer; multiply the generated scaling factor element-wise with the processed frame-level features to obtain the weighted features of the corresponding single-frame face image; and aggregate the weighted frame-level features of the single-frame face images within the corresponding video sub-segment to generate the aggregated video features of the corresponding video sub-segment.

[0073] In the example, the emotion description generation and classification unit 503 can be configured to: map the sequence of text into sentence-level feature vectors; input the sentence-level feature vectors in parallel into a classification head and a regression head to output the emotion category and corresponding emotion intensity value of each video segment. The classification head includes a fully connected layer and a Softmax activation function to transform the sentence-level feature vectors into a probability distribution corresponding to a predefined emotion and take the emotion corresponding to the one with the highest probability as the emotion category of the corresponding video segment. The regression head includes a fully connected layer and a Sigmoid activation function to transform the sentence-level feature vectors into continuous values ​​within a predetermined range as the emotion intensity value of the corresponding video segment.

[0074] The above combination Figures 1 to 4 The specific operations shown are respectively by Figure 5 The corresponding unit in the illustrated emotion detection device 500 performs this function; specific operational details will not be elaborated here. By employing the emotion detection device according to the exemplary embodiments of this disclosure, it is possible to effectively adapt pre-trained visual-language models without sacrificing performance, enhance their temporal modeling capabilities specifically for dynamic facial expression recognition tasks, and provide multi-dimensional and highly interpretable outputs.

[0075] Figure 6 This is a block diagram illustrating a computing system including at least one computing device and at least one storage device of storage instructions according to an exemplary embodiment of the present disclosure.

[0076] like Figure 6 As shown, the computing system 600 according to an exemplary embodiment of the present invention includes a computing device 601 and a storage device 602. The storage device 602 stores computer-executable instructions. When the computer-executable instructions are executed by the computing device 601, the emotion detection method described in any of the foregoing embodiments is executed.

[0077] The computing device 601 can be deployed in a server or client, or on a node device in a distributed network environment. Furthermore, the computing device 601 can be a PC, tablet, personal digital assistant, smartphone, web application, or other device capable of executing the aforementioned set of instructions. Here, the computing device is not necessarily a single computing device; it can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The computing device can also be part of an integrated control system or system manager, or can be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission). In the computing device, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor also includes analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0078] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores instructions, which, when executed by at least one computing device, cause the at least one computing device to perform the emotion detection method described in any of the foregoing embodiments. The computer-readable storage medium includes magnetic media such as floppy disks and magnetic tapes, optical media (including optical disc (CD) ROMs and DVD ROMs), magneto-optical media such as floppy discs, hardware devices such as ROMs and RAMs designed for storing and executing program commands, and flash memory. The instructions may include language code executable by a computer using an interpreter and machine language code generated by a compiler.

[0079] By employing this disclosure, it is possible to accurately capture and analyze the dynamic changes in facial emotions in videos (including, for example, the type, intensity, and timing of the emotion), while ensuring computational efficiency and improving the interpretability of the results.

[0080] The processes, methods, or algorithms disclosed herein can be transmitted to, or implemented by, a processing device, controller, or computer, which may include any existing programmable electronic control unit or a dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored in various forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media (such as ROM devices) and information variablely stored on writable storage media (such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media). The processes, methods, or algorithms can also be implemented in a software executable object. Optionally, the processes, methods, or algorithms can be implemented wholly or partially using suitable hardware components (such as ASICs, FPGAs, state machines, controllers, or other hardware components or devices) or a combination of hardware components, software components, and firmware components.

[0081] Although this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered merely for descriptive purposes and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or if components in the described system, architecture, apparatus, or circuit are replaced or supplemented with other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents shall be construed as included in this disclosure.

Claims

1. An emotion detection method, characterized in that, The emotion detection method includes: The input video is standardized frame by frame to obtain multiple single-frame face images as the output video. The single-frame face image is processed by a visual encoder to extract frame-level features of the single-frame face image. Identify the time points of emotional transition in the output video, and segment the output video based on the time points to obtain multiple video sub-segments; For each video segment, the frame-level features of the single-frame face image within the corresponding video segment are aggregated using a temporal dynamic adapter to generate the aggregated video features of the corresponding video segment. The aggregated video features of the corresponding video segments are input into a large language model to generate text describing emotional states; The text is analyzed using a text classification regression model to output the emotion category and corresponding emotion intensity value for each video segment.

2. The emotion detection method according to claim 1, characterized in that, The input video is a video clip containing a face or a real-time video stream.

3. The emotion detection method according to claim 1, characterized in that, The standardized processing includes face detection, feature point extraction, face alignment, and cropping operations.

4. The emotion detection method according to claim 1, characterized in that, The step of processing the single-frame face image using a visual encoder to extract frame-level features of the single-frame face image includes: uniformly dividing the single-frame face image into multiple non-overlapping image blocks; flattening and linearly projecting the multiple non-overlapping image blocks to generate an initial feature sequence; and generating frame-level features of the single-frame face image based on the initial feature sequence using a transformer, wherein the transformer includes one or more transformer layers connected in sequence, each transformer layer including a multi-head self-attention layer and a lightweight shared adapter connected to the multi-head self-attention layer.

5. The emotion detection method according to claim 1, characterized in that, The steps for identifying the time points of emotional transition in the output video include: Determine the center feature vector of the feature vectors of the frame-level features of multiple single-frame face images included within the sliding time window; The feature variation dispersion score of the feature vector of the frame-level feature of the plurality of single-frame face images is determined, wherein the feature variation dispersion score is the average cosine distance from the feature vector of the frame-level feature of the corresponding single-frame face image to the center feature vector. In response to the feature vectors whose feature change dispersion scores within the sliding time window are greater than a predetermined threshold, the single-frame face images corresponding to these features constitute a continuous sequence, and the number of single-frame face images included in the continuous sequence is greater than or equal to a predetermined number, the start time of the continuous sequence is determined as the time point.

6. The emotion detection method according to claim 1, characterized in that, The temporal dynamic adapter includes a gated recurrent unit and a dynamic scaler. The dynamic scaler is configured to: for each single-frame face image, generate a corresponding scaling factor based on the frame-level features processed by the gated recurrent unit through a learnable linear transformation layer; multiply the generated scaling factor element-wise with the processed frame-level features to obtain the weighted features of the corresponding single-frame face image; and aggregate the weighted frame-level features of the single-frame face images within the corresponding video sub-segment to generate the aggregated video features of the corresponding video sub-segment.

7. The emotion detection method according to claim 1, characterized in that, The steps of analyzing the text using a text classification and regression model to output the emotion category and corresponding emotion intensity value of each video segment include: mapping the text sequence into sentence-level feature vectors; inputting the sentence-level feature vectors in parallel into a classification head and a regression head to output the emotion category and corresponding emotion intensity value of each video segment, wherein the classification head includes a fully connected layer and a Softmax activation function to transform the sentence-level feature vectors into a probability distribution corresponding to a predefined emotion and taking the emotion corresponding to the highest probability as the emotion category of the corresponding video segment; and wherein the regression head includes a fully connected layer and a Sigmoid activation function to transform the sentence-level feature vectors into continuous values ​​within a predetermined range as the emotion intensity value of the corresponding video segment.

8. An emotion detection device, characterized in that, The emotion detection device includes: The video preprocessing unit is configured to perform frame-by-frame normalization on the input video to obtain multiple single-frame face images as the output video. The dynamic temporal feature extraction and aggregation unit is configured to process the single-frame face image through a visual encoder to extract the frame-level features of the single-frame face image, identify the time points of emotion transition in the output video, and segment the output video based on the time points to obtain multiple video sub-segments. For each video sub-segment, the frame-level features of the single-frame face image in the corresponding video sub-segment are aggregated through a temporal dynamic adapter to generate the aggregated video features of the corresponding video sub-segment. The emotion description generation and classification unit is configured to input the aggregated video features of the corresponding video sub-segments into a large language model to generate text describing the emotional state, and to analyze the text using a text classification and regression model to output the emotion category and corresponding emotion intensity value for each video sub-segment.

9. A computing system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the emotion detection method according to any one of claims 1-7.

10. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the emotion detection method according to any one of claims 1-7.

Images