Teenager multi-mode emotion recognition method and system integrating dynamic time sequence optimization and BiLSTM

Abstract

The invention belongs to the cross technical field of computer vision, emotion calculation, machine learning and pattern recognition, and particularly relates to a teenager multi-modal emotion recognition method and system integrating dynamic time sequence optimization and BiLSTM. The method comprises the following steps: acquiring a multi-modal initial data set of to-be-recognized adolescent emotions; based on dynamic time sequence optimization and BiLSTM, constructing a teenager multi-modal emotion recognition model; and obtaining a teenager multi-modal emotion recognition result based on the multi-modal initial data set and the teenager multi-modal emotion recognition model. According to the method, through the characteristic of coping with changeable emotions of the teenagers through time sequence optimization, the defect of a traditional method in capturing rapidly changing emotions is made up, the difference caused by the age span of the teenagers is solved through dynamic weight adaptation, and the accuracy of emotion recognition is improved by supplementing physiological data.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary fields of computer vision, emotion computing, machine learning and pattern recognition, and specifically relates to a method and system for multimodal emotion recognition of adolescents that integrates dynamic temporal optimization and BiLSTM. Background Technology

[0002] In today's global digital wave, video emotion recognition technology is penetrating numerous fields such as human-computer interaction and mental health monitoring at an unprecedented pace, becoming an important bridge connecting humans and machines in emotional communication. However, when focusing on adolescents, the application of existing technologies encounters significant bottlenecks. Most current mainstream general-purpose video emotion recognition models rely on datasets from a wide range of populations for training, and their generalization ability struggles to accurately capture the unique emotional expression characteristics of adolescents, often falling into the trap of "adult-like" misjudgments—for example, incorrectly classifying exaggerated expressions characteristic of adolescents during puberty or changes in tone of voice during voice change as negative emotions, directly leading to a significant decline in recognition accuracy.

[0003] The unique characteristics of adolescents are particularly prominent: the 12-18 age range precisely covers the critical developmental stage of puberty. During this period, adolescents exhibit significant individual differences in physiological development (such as rapid changes in facial features and vocal cord maturation) and psychological maturity, leading to marked differentiation in emotional expression patterns across different age groups. Simultaneously, influenced by dramatic fluctuations in hormone levels, their emotions often display a high-frequency fluctuation characteristic of "sudden rises and falls," specifically manifested in extremely short durations of micro-expressions, rapid changes in vocal tone, and rapid fluctuations in physiological indicators accompanying emotional changes. Faced with these dynamic characteristics, traditional single-modal recognition methods or fixed-time-sequence analysis models are clearly insufficient to comprehensively and accurately capture the true emotional state of adolescents.

[0004] Therefore, developing a multimodal video-based dynamic emotion recognition method specifically for adolescents aged 12-18 has become an urgent need for the application of current technology in the field of adolescent mental health. This method needs to overcome the generalization limitations of generic models, deeply adapt to the age range and emotional fluctuation characteristics of adolescents, and integrate multimodal data such as facial micro-expressions, audio signals, and physiological temperature. By combining dynamic temporal optimization and intelligent feature fusion, it can achieve real-time and accurate identification of adolescent emotions, thereby providing solid technical support for practical applications such as adolescent mental health monitoring and personalized guidance in educational scenarios. Summary of the Invention

[0005] To address the problems of existing technologies, this invention provides a method and system for multimodal emotion recognition in adolescents that integrates dynamic temporal optimization and BiLSTM. The aim is to address the volatile nature of adolescent emotions through temporal optimization, overcoming the shortcomings of traditional methods in capturing rapidly changing emotions and enabling more sensitive tracking of dynamic emotional changes. Dynamic weight adaptation addresses the differences arising from the age range of adolescents, avoiding the use of a uniform standard for all age groups and better adapting to the emotional expression characteristics of different developmental stages. Supplementing physiological data improves the accuracy of emotion recognition, forming a multi-dimensional judgment basis and overcoming the limitations of relying solely on external features, making emotion recognition more accurate, especially in effectively identifying those masked emotions.

[0006] To achieve the above objectives, the present invention provides the following solution: A method for adolescent multimodal emotion recognition integrating dynamic temporal optimization and BiLSTM, the method comprising: Obtain a multimodal initial dataset of adolescent emotions to be identified; A multimodal emotion recognition model for adolescents is constructed based on dynamic temporal optimization and BiLSTM. Based on the multimodal initial dataset and the adolescent multimodal emotion recognition model, the adolescent multimodal emotion recognition results were obtained.

[0007] Preferred methods for obtaining a multimodal initial dataset of adolescent emotions to be identified include: Obtain the initial dataset, which includes a high-resolution facial video stream for extracting micro-expressions, an audio stream as audio data, and facial temperature data acquired by an infrared thermal imager. The initial dataset was labeled to obtain a multimodal initial dataset of adolescent emotions.

[0008] Preferably, the adolescent multimodal emotion recognition model includes: a dynamic temporal optimization module, a multimodal feature extraction module, a BiLSTM module, a feature dynamic weighting module, and a temporal causal attention module; The dynamic time series optimization module is used to adaptively adjust the time granularity according to emotional fluctuations to achieve multimodal data alignment; The multimodal feature extraction module is used to extract multimodal features based on aligned multimodal data. The BiLSTM module is used to input the extracted multimodal features into the BiLSTM network to obtain the hidden state sequence of each modality feature. The feature dynamic weight module is used to obtain the dynamic weights of each modal feature and combine them with the hidden state sequence of each modal feature to obtain the multimodal fusion hidden state sequence. The temporal causal attention module is used to perform causal attention calculations on the multimodal fusion hidden state sequence to obtain the temporal causal enhanced hidden state.

[0009] Preferably, the dynamic time-series optimization module adaptively adjusts the time granularity based on emotional fluctuations to achieve multimodal data alignment, including the following methods: Construct a timeline with a preset precision, and map the original timestamps of each modality in the initial multimodal dataset onto the timeline; Using the time axis as a reference, the data of each modality is initially framed according to a preset time interval to obtain an initial frame sequence; Based on the initial frame sequence, obtain image differences and audio differences; Based on image and audio differences, the initial frame sequence is dynamically adjusted in two dimensions to obtain the adjusted data frame sequence. The adjusted data frame sequence is preprocessed to achieve multimodal data alignment.

[0010] Preferably, the method for extracting multimodal features based on aligned multimodal data in the multimodal feature extraction module includes: Based on the Dlib library, facial key points are extracted from high-definition facial video streams, displacement and velocity are calculated, dynamic features are obtained through standardization and PCA dimensionality reduction, and intensity features are obtained by combining the FACS system to mark the contraction intensity of facial muscles. The dynamic features and intensity features are then combined to obtain micro-expression features. Multiple indicators of a preset area are extracted from facial temperature data to obtain temperature features. Facial temperature data is then input into a 3-layer MLP model to obtain emotional activation. The temperature features and emotional activation are then combined to obtain physiological features. The audio stream is subjected to a short-time Fourier transform and then converted by a Mel filter bank. The Mel frequency cepstral coefficients are obtained by taking the logarithm and discrete cosine transform. Combined with preset acoustic parameters, the audio characteristics are obtained.

[0011] Preferably, the method for inputting the extracted multimodal features into the BiLSTM network to obtain the hidden state sequence of each modality feature includes: The extracted modal features are input into the corresponding BiLSTM network to obtain the hidden state sequence of each modal feature; The BiLSTM network consists of two LSTM networks, a forward network and a backward network, which capture temporal dependencies from the forward and backward directions of the feature sequence, respectively.

[0012] Preferably, the method for obtaining the dynamic weights of each modal feature by the feature dynamic weight module and combining them with the hidden state sequences of each modal feature to obtain the multimodal fusion hidden state sequence includes: Construct an age-weighted network and obtain the dynamic weights of each modality feature based on the age-weighted network; The hidden state sequences of each modality feature are weighted and summed with the dynamic weights of each modality feature to obtain the multimodal fusion hidden state sequence.

[0013] Preferably, the method for the temporal causal attention module to perform causal attention calculation on the multimodal fused hidden state sequence to obtain the temporally causal enhanced hidden state includes: The multimodal fusion hidden state sequence is preprocessed to obtain standardized features; Causal attention calculation is performed on standardized features to obtain attention weights; The attention weights are weighted and summed to obtain the initial fusion hidden state. The initial fusion hidden state and the multimodal fusion hidden state sequence are added together and normalized to obtain the temporal causal reinforcement hidden state.

[0014] Preferably, methods for obtaining adolescent multimodal emotion recognition results based on a multimodal initial dataset and an adolescent multimodal emotion recognition model include: The initial multimodal dataset is input into the adolescent multimodal emotion recognition model to obtain the latent state after temporal causal reinforcement. The latent state, enhanced with temporal causality, is input into a classification head containing two fully connected layers to obtain a sentiment weight vector. L1 normalization was performed on the emotion weight vector to obtain the multimodal emotion recognition results for adolescents.

[0015] The present invention also provides a multimodal emotion recognition system for adolescents that integrates dynamic temporal optimization and BiLSTM. The system is used to implement the aforementioned method and includes: a data acquisition module, a model building module, and an emotion recognition module. The data acquisition module is used to acquire a multimodal initial dataset of adolescent emotions to be identified; The model building module is used to build a multimodal emotion recognition model for adolescents based on dynamic temporal optimization and BiLSTM. The emotion recognition module is used to obtain multimodal emotion recognition results for adolescents based on the multimodal initial dataset and the adolescent multimodal emotion recognition model.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention addresses the volatile emotions of adolescents through time-series optimization: a dynamic time-series optimization mechanism is designed to flexibly adjust the time granularity of analysis according to the intensity of emotional fluctuations—when emotional fluctuations are large, more refined time units are used to capture details; when emotions are calm, time units are appropriately merged to reduce redundancy, while ensuring that different types of data are accurately aligned in time. This mechanism effectively compensates for the shortcomings of traditional methods in capturing rapidly changing emotions and can more sensitively track the dynamic changes of emotions.

[0017] (2) This invention solves the differences caused by the age range of adolescents through dynamic weight adaptation: 12-18 years old is a critical stage of growth, and adolescents of different ages have obvious differences in emotional expression. This invention introduces a dynamic weight mechanism, which can automatically adjust the importance of different features such as facial expressions, voice, and physiological data in emotion recognition according to specific age, avoid using a uniform standard to process all age groups, and better adapt to the emotional expression characteristics of different growth stages.

[0018] (3) This invention improves the accuracy of emotion recognition by supplementing physiological data: Traditional emotion recognition relies heavily on external features such as facial expressions or voice, while teenagers sometimes deliberately conceal their true emotions, leading to misjudgments. This invention incorporates physiological data (such as changes in facial temperature), which is more difficult to fake and can reflect the internal changes in emotions. Combining physiological data with external facial expressions and voice features forms a multi-dimensional basis for judgment, making up for the limitations of relying solely on external features, and making emotion recognition more accurate, especially for those concealed emotions. Attached Figure Description

[0019] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a simplified flowchart of the multimodal emotion recognition method for adolescents integrating dynamic temporal optimization and BiLSTM, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the MLP model framework according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the audio feature extraction process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the LSTM structure according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the BiLSTM structure according to an embodiment of the present invention; Figure 6This is a simplified flowchart of the temporal causal attention module according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the overall framework of an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1 like Figure 1 As shown, this invention provides a method for multimodal emotion recognition in adolescents that integrates dynamic temporal optimization and BiLSTM, including: Obtain a multimodal initial dataset of adolescent emotions to be identified; A multimodal emotion recognition model for adolescents is constructed based on dynamic temporal optimization and BiLSTM. Based on the multimodal initial dataset and the adolescent multimodal emotion recognition model, the adolescent multimodal emotion recognition results were obtained.

[0024] Furthermore, the specific implementation process of this invention is as follows: Methods for obtaining a multimodal initial dataset of adolescent emotions to be identified include: Obtain the initial dataset, which includes a high-resolution facial video stream for extracting micro-expressions, an audio stream as audio data, and facial temperature data acquired by an infrared thermal imager.

[0025] This invention focuses on adolescents aged 12-18. Data collection utilized eight representative scenarios, including classrooms, homes, and social gatherings, with varying video durations. The collected data includes multiple types of information: first, high-definition facial video streams for micro-expression extraction, with preset standards for resolution and frame rate; second, audio streams as audio data, with preset sampling rates and quantization parameters; and third, facial temperature data collected by an infrared thermal imager, with preset standards for resolution and temperature measurement range, used as physiological data. In terms of age distribution, each of the four key age groups for adolescent development is represented in a quarter of the data, while maintaining a balanced 1:1 male-to-female ratio.

[0026] The initial dataset was labeled to obtain a multimodal initial dataset of adolescent emotions.

[0027] The video annotation was conducted by three experts with experience in adolescent psychology research. They labeled every 5-second segment of the video according to eight emotion categories. The video included audio streams as audio data and facial temperature data collected by an infrared thermal imager. Cross-comparison was used to ensure the annotation consistency rate met the standards, ultimately forming a video dataset where the three types of data are precisely matched with millisecond-level timestamps.

[0028] Furthermore, based on dynamic temporal optimization and BiLSTM, a multimodal emotion recognition model for adolescents is constructed. The multimodal emotion recognition model for adolescents includes: a dynamic temporal optimization module, a multimodal feature extraction module, a BiLSTM module, a feature dynamic weighting module, and a temporal causal attention module. The dynamic time series optimization module is used to adaptively adjust the time granularity according to emotional fluctuations to achieve multimodal data alignment; The multimodal feature extraction module is used to extract multimodal features based on aligned multimodal data. The BiLSTM module is used to input the extracted multimodal features into the BiLSTM network to obtain the hidden state sequence of each modality feature. The feature dynamic weight module is used to obtain the dynamic weights of each modal feature and combine them with the hidden state sequence of each modal feature to obtain the multimodal fusion hidden state sequence. The temporal causal attention module is used to perform causal attention calculations on the multimodal fusion hidden state sequence to obtain the temporal causal enhanced hidden state.

[0029] Specifically, the dynamic time-series optimization module adaptively adjusts the time granularity based on emotional fluctuations to achieve multimodal data alignment. This includes the following methods: Construct a timeline: Construct a timeline with a preset precision, and map the original timestamps of micro-expression, physiological, and audio data onto this timeline to ensure that the initial time points of these three types of data are consistent and that the error is controlled within a preset range.

[0030] Frame segmentation: Based on the time axis, the modal data is initially segmented into frames according to a preset time interval to obtain an initial frame sequence; Calculate image differences: After initial framing, the frame difference method is used to calculate the differences between the images for the obtained video frames, thereby capturing the changes between frames.

[0031] Frame differencing is a technique that detects dynamic changes by comparing pixel differences between consecutive frames in a video. For two consecutive frames... (Current frame) and (Previous frame), Difference map The calculation method and video pixel change rate are as follows: ; Video pixel change rate = .

[0032] Calculate audio differences: For audio data, it is first divided into audio segments according to the initial frame interval, and the energy value of each audio segment is calculated (by summing the squares of the audio signals and taking the average). Then, the ratio of the energy difference between adjacent audio segments to the energy value of the previous segment is calculated to obtain the audio energy change rate, which is used as a measure of audio change.

[0033] Calculate the rate of change of audio energy, given the energy values ​​of two consecutive frames. and The calculation formula is as follows: ; Audio energy change rate = .

[0034] in: N: Represents the number of sampling points in a frame.

[0035] n : This is the sample index within the frame, and its value range is... n =0,1,…, N -1.

[0036] : indicates the first t The first frame n Each sample value. Subscript t Indicates the frame index, so It is a discrete-time signal in a frame t Data within.

[0037] t : Indicates the frame index. t =0,1,2,…

[0038] Two-dimensional dynamic frame segmentation adjustment: This is based on the video pixel change rate and audio energy change rate. When both are at a high level (indicating drastic emotional fluctuations), the initial frame segmentation is subdivided into shorter frames. When both are at a low level (indicating calmer emotions), two consecutive initial frame segments are merged into a longer frame (with different proportions of the preceding and following segments). When both are at an intermediate level, the initial frame segmentation duration is maintained. Simultaneously, time deviations are corrected through cross-modal verification to ensure frame sequence synchronization. Through this two-dimensional dynamic frame segmentation adjustment, an adjusted data frame sequence is obtained. Data preprocessing: During data preprocessing, video frames in the data frame sequence are adjusted to a preset resolution, audio undergoes denoising and L2 normalization, and physiological data is segmented according to video frame granularity and time-calibrated to achieve millisecond-level alignment of the three types of data.

[0039] L2 normalization is the process of normalizing each element of a vector ( x Divide it by its L2 norm so that the length of the final vector is 1 (unit vector).

[0040] Given vector for: ; in, Given a vector Elements in; The L2 norm is: ; L2 normalized vector for: .

[0041] The dynamic temporal optimization module adaptively adjusts the time granularity based on emotional fluctuations and aligns multi-source data, improving the sensitivity and recognition accuracy of the adolescent multimodal emotion recognition model to dynamic changes in adolescent emotions. Specifically, the multimodal feature extraction module uses methods to extract multimodal features from aligned multimodal data, including: Micro-expression feature extraction: Facial key points are extracted from high-definition facial video streams using the Dlib library. The displacement and velocity are calculated, and dynamic features are obtained through Z-score normalization and PCA dimensionality reduction. The intensity of facial muscle contraction is marked by the FACS system to obtain intensity features. The dynamic features and intensity features are then combined to obtain micro-expression features.

[0042] The process of calculating displacement and velocity is as follows: (1) Calculation of displacement The displacement is the Euclidean distance relative to the neutral frame. For the The first frame Key points: ; in, It is the first The first frame The coordinates of the key points It is the first The coordinates of a point in a neutral frame.

[0043] (2) Speed ​​calculation Velocity is the rate of change of displacement between adjacent frames.

[0044] For the The first frame Key points: ; in It is the frame time interval.

[0045] The dynamic features and intensity features are combined to obtain micro-expression features. The merging process is as follows: The merging process first performs time alignment to ensure that dynamic features (obtained by PCA dimensionality reduction based on the displacement and velocity of all facial key points) and intensity features (based on the contraction intensity of specific muscle groups calculated by the FACS system) from the same time period correspond precisely at the frame level. Then, the two types of feature vectors are Z-score standardized to eliminate dimensional differences. Finally, the standardized dynamic feature vector and intensity feature vector are directly concatenated along the feature dimension to form a joint feature vector that integrates global motion patterns and local muscle activity intensity, thereby obtaining the final micro-expression features.

[0046] The Dilb library is a modern toolkit widely used in machine learning, image processing, and computer vision tasks.

[0047] Z-score normalization is a data preprocessing technique used to transform data at different scales into a form with zero mean and unit variance.

[0048] PCA dimensionality reduction is a statistical method used to simplify the dimensionality of a dataset while preserving as much of the main variation information as possible from the original data. PCA transforms a set of potentially related variables into a set of linearly independent variables—called principal components—through linear transformation. The first principal component has the largest variance, and each subsequent component has the largest variance provided it is orthogonal to all preceding components.

[0049] The FACS system is a standardized human facial expression analysis system. It breaks down facial movements into the smallest motion units (called Action Units), and through the combination of these units, it can accurately describe any facial expression.

[0050] Physiological feature extraction: Multiple indicators (quantified data on average temperature and temperature difference) of preset regions are extracted from facial temperature data (such as the forehead and left and right cheekbones) to obtain temperature features. These temperature features are then input into a 3-layer MLP model (e.g., Figure 2The temperature feature vector is obtained by combining the temperature feature vector with the emotional activation vector (scalar or low-dimensional vector) output by the 3-layer MLP. The extracted temperature feature vector is then directly combined with the emotional activation vector (scalar or low-dimensional vector) output by the 3-layer MLP through vector concatenation to obtain physiological features that reflect physiological changes related to emotions.

[0051] Audio feature extraction: like Figure 3 As shown, the audio stream is first calculated using the STFT (Short Time Fourier Transform) and then transformed using a Mel filter bank. After taking the logarithm and performing the DCT (Discrete Cosine Transform), the MFCC (Mel Frequency Cepstral Coefficients) is obtained. Multiple preset acoustic parameters such as the spectral centroid, bandwidth, spectral entropy, energy characteristics, and harmonic-to-noise ratio are then added to form a fixed-dimensional feature sequence for each frame, thus obtaining the audio features.

[0052] Multimodal feature extraction can uncover cross-modal correlations, thereby improving the ability of adolescent multimodal emotion recognition models to understand complex information.

[0053] Specifically, the BiLSTM module inputs the extracted multimodal features into the BiLSTM network to obtain the hidden state sequence of each modality feature. The methods include: The extracted micro-expression features, physiological features, and audio features are input into three independent BiLSTM networks, each containing two hidden layers (half forward and half backward), with dropout rates set between layers. The BiLSTM networks output the hidden state sequences of each modality feature, with all three having the same dimension, where T is the total number of frames.

[0054] like Figures 4-5 As shown, the BiLSTM network consists of two LSTM networks, a forward network and a backward network, which capture temporal dependencies from the forward and backward directions of the feature sequence, respectively.

[0055] Figure 4 This demonstrates the internal working principle of an LSTM unit, with the symbols explained below: : Short-term memory / output from the previous moment.

[0056] Long-term memory of the previous moment.

[0057] : Input data from the previous moment.

[0058] : Short-term memory / output at the current moment.

[0059] Long-term memory updated at the current moment.

[0060] : Input data at the current moment.

[0061] : Short-term memory / output in the next moment.

[0062] Long-term memory of the next moment.

[0063] : Input data for the next moment.

[0064] σ (Sigmoid): A gated switch that outputs a number between 0 and 1, determining "how much to keep" or "how much to discard".

[0065] tanh: Generates new information, outputting a number between -1 and 1 as a candidate memory.

[0066] Multiplication operation is used to regulate the flow of information using gating signals.

[0067] Addition is an operation used to add new information to long-term memory.

[0068] Figure 5 Explanation of symbols in the text: , , : Represents the input sequence, which are the input data at time t-1 (previous), t (current), and t+1 (next).

[0069] : The hidden state generated by the forward LSTM at time t.

[0070] : The hidden state generated by the inverse LSTM at time t.

[0071] The BiLSTM module can improve the accuracy and robustness of emotion recognition in adolescent multimodal emotion recognition models.

[0072] Specifically, the method for obtaining the dynamic weights of each modality feature by the feature dynamic weight module, and combining them with the hidden state sequences of each modality feature to obtain the multimodal fusion hidden state sequence includes: Constructing an age-weighted network: The age-weighted network is a two-layer fully connected structure with 64 hidden nodes each, employing the Tanh activation function. The input consists of age information from adolescents aged 12-18, represented using one-hot encoding. Trained on a large dataset of adolescent emotion recognition data, the age-weighted network learns the differences in sensitivity to micro-expressions, physiological features, and audio characteristics across different age groups, outputting dynamic weights for these three features, with the sum of these dynamic weights equal to 1.

[0073] One-hot coding is a technique for converting discrete categorical data into binary vectors.

[0074] (1) The input is the one-hot code for the age group. g These are category labels; they are 5-dimensional. one - hot The encoding vector is given by the following formula: ; (2) Construct an age weight network with 5 nodes in the input layer corresponding to 5 age groups, 64 nodes in the hidden layer using the Tanh activation function, and 3 nodes in the output layer corresponding to the dynamic weights of three types of features: micro-expression, physiological and audio. x For input, the formula is: ; (3) Output the weighting coefficients of the three types of features (micro-expression features, physiological features, and audio features) so that the adolescent multimodal emotion recognition model can dynamically adjust the feature fusion weights according to the emotional expression characteristics of adolescents of different age groups.

[0075] Weighted fusion hidden state: The micro-expression hidden state sequence, physiological hidden state sequence, and audio hidden state sequence output by the BiLSTM network are multiplied by the corresponding weights output by the age-weighted network. Then, the three weighted hidden state sequences are summed to obtain a multimodal fusion hidden state sequence suitable for the adolescent's age. This dynamic weighting mechanism allows the model to automatically adjust the importance of each modality feature in emotion recognition based on the characteristics of different age groups, improving the adaptability of the multimodal emotion recognition model for adolescents of different age groups.

[0076] The feature dynamic weighting module enables the adolescent multimodal emotion recognition model to adaptively focus on the importance of features in different age groups, thereby improving accuracy.

[0077] Specifically, such as Figure 6 As shown, the temporal causal attention module performs causal attention computation on the multimodal fused hidden state sequence to obtain the temporally causal enhanced hidden state, including the following methods: The input multimodal fusion hidden state sequence is preprocessed to determine the time steps (i.e., video frames) and feature dimensions. Layer normalization is then applied to the multimodal fusion hidden state sequence. By calculating the mean and variance of the features, the sequence is transformed into standardized features with a mean of 0 and a variance of 1. This stabilizes the subsequent training process and prevents excessively large differences in feature values ​​from negatively impacting the performance of the adolescent multimodal emotion recognition model.

[0078] Perform causal attention calculation on the standardized features to obtain attention weights: First, learnable weight matrices are used to generate three types of vectors: query, key, and value. The correlation matrix is ​​a parameter matrix that can be optimized through training. The dimensions of queries and keys are set to 128, and the dimension of values ​​is also set to 128 to balance computational efficiency and feature representation capability. Next, the correlation strength between frames is calculated, and the attention score matrix is ​​obtained through a formula. Then, a causal mask (upper triangular matrix) is applied to ensure that the current frame only focuses on the current and previous frames, preventing information from future frames from interfering with the judgment of the current frame. Finally, the attention score matrix is ​​normalized to obtain the attention weight distribution among the frames.

[0079] (1) Calculate the attention score matrix Objective: To calculate the association strength between all frame pairs.

[0080] formula: ; Where S is the attention score matrix, Q is the query vector matrix, obtained by linear transformation of the input sequence, K is the key vector matrix, obtained by linear transformation of the input sequence, d is the dimension of the query and key vectors (128), and T is the number of time steps.

[0081] (2) Applying causal masking Objective: To ensure that the current frame can only focus on the current and past frames.

[0082] formula: ; Where S is the attention score matrix and M is the causal mask.

[0083] (3) Calculate attention weights Objective: To convert association strength into a normalized attention distribution.

[0084] formula: ; Where A is the attention weight matrix, It is the first Frame to the first The attention weight of a frame, where T is the number of time steps.

[0085] The attention weights are weighted and summed to obtain the initial fused hidden state. To prevent information loss, the residual connections are preserved. The initial fused hidden state and the original multimodal fused hidden state sequence are added together and then layer normalization is performed to finally obtain the hidden state after temporal causal reinforcement.

[0086] The temporal causal attention module ensures that emotion recognition follows causal logic over time, thereby strengthening the correlation between key events and emotional changes and improving the accuracy and interpretability of the multimodal emotion recognition model for adolescents.

[0087] Furthermore, based on the multimodal initial dataset and the adolescent multimodal emotion recognition model, methods for obtaining adolescent multimodal emotion recognition results include: The initial multimodal dataset is input into the adolescent multimodal emotion recognition model to obtain the latent state after temporal causal reinforcement. L1 normalization was performed on the emotion weight vector to obtain the multimodal emotion recognition results for adolescents.

[0088] Specifically: By inputting the latent state enhanced with temporal causality into a classification head containing two fully connected layers, a sentiment weight vector is obtained: The latent state after temporal causality enhancement is input into a classification head containing two fully connected layers. After processing, the output is an emotion weight vector. Then, L1 normalization is applied to make the sum of the weights equal to 1.0, thus obtaining the proportion of each emotion, which is the result of adolescent multimodal emotion recognition.

[0089] L1 normalization (also known as Manhattan normalization or absolute value normalization) refers to dividing the feature vector of each sample by its L1 norm (i.e., the sum of the absolute values ​​of all elements in the feature vector) so that the L1 norm of the normalized vector is equal to 1. The difference between L1 normalization and the previous L2 normalization is that different norms are used to scale the data.

[0090] The L1 norm is: .

[0091] Multi-label emotion output transforms processed features into specific emotion labels to accurately identify and output the emotional state expressed by the input data.

[0092] The overall framework of this invention is as follows: Figure 7 As shown, the video dataset (including high-definition facial videos and audio streams from which micro-expressions are extracted) is first subjected to preliminary frame segmentation and dynamic adjustment of the frame segmentation, while the physiological dataset is processed simultaneously. Then, micro-expression features are extracted using Dlib+FACS, physiological features are extracted using MLP, and audio features are extracted using STFT+DCT+MFCC. Subsequently, the three types of features are input into BiLSTM for processing, and weighted fusion is performed by combining the output of the age-weighted network. Finally, the results are output through the temporal causal attention module.

[0093] In summary, this invention proposes an innovative method for dynamic emotion recognition in multimodal videos of adolescents. This method cleverly integrates a dynamic temporal frame optimization module with a three-branch BiLSTM multimodal fusion network, significantly improving the recognition ability, adaptability, and accuracy of multimodal emotion recognition models for adolescents in scenarios with fluctuating emotions and a wide age range. This method utilizes dynamic temporal frame optimization to process multimodal data, combined with age-weighted feature representations, to fuse multidimensional emotional information and age adaptability, thus optimizing the feature extraction and fusion process.

[0094] The three-branch BiLSTM enables the adolescent multimodal emotion recognition model to capture the temporal dependencies of micro-expressions, physiological features, and audio features, while the temporal causal attention mechanism strengthens the learning of the logic of emotion development by reinforcing the "event-emotion" association. The two work together to overcome the limitations of single-modality and temporal modeling. In addition, the dynamic weight adaptation mechanism ensures that the model can adapt to the emotional expression characteristics of adolescents of different ages, not only paying attention to the subtle changes in single-modal features, but also grasping the synergistic association of multimodal features.

[0095] This invention provides an efficient and accurate solution for adolescent emotion recognition by integrating multimodal data preprocessing, age-adapted feature fusion strategies, and a temporal causal reinforcement deep learning architecture. It can address the complex challenges of adolescent emotion recognition, meet real-time and accuracy requirements, and demonstrates broad practical application potential.

[0096] Example 2 This embodiment illustrates the method described in the foregoing embodiment using specific data: Step 1: Data Acquisition. For adolescents aged 12-18, eight typical scenarios were selected, including classroom, home, and social gatherings, and 5-30 minutes of video data were collected. This included high-definition facial videos at 1920×1080 resolution and 30fps (for micro-expression extraction), audio streams at 44.1kHz sampling rate and 16-bit quantization (as audio data), and facial temperature data collected using an infrared thermal imager (320×240 resolution, 32-40℃ temperature measurement range) (as physiological data). In terms of age distribution, 20% were allocated to each of the following age groups: 12-13, 13-14, 14-15, 15-16, and 17-18, ensuring coverage of key stages of adolescent development (male-to-female ratio 1:1).

[0097] Step 2: Video Annotation. Three experts with experience in adolescent psychology research annotated every 5-second segment of the video according to eight emotion categories, including happiness and irritability. Cross-referencing ensured an annotation consistency rate of ≥85%, ultimately forming a video dataset. The three data categories were precisely matched using millisecond-level timestamps.

[0098] Step 3: Construct a timeline. Construct a timeline with an accuracy of 0.1 seconds, mapping the original timestamps of micro-expression, physiological, and audio data onto this timeline to ensure that the initial time points of the three types of data are consistent, with the error controlled within 0.05 seconds.

[0099] Step 4: Frame segmentation. Based on the timeline, perform initial frame segmentation at 1-second intervals to obtain the initial frame sequence.

[0100] Step 5: Calculate image differences. For the video frames after initial framing, the frame difference method is used to calculate image differences.

[0101] Step 6: Calculate audio differences. The audio data is first divided into segments according to the initial frame interval. The energy value of each segment is calculated (the average value is obtained by summing the squares of the audio signals). Then, the ratio of the energy difference between adjacent segments to the energy value of the previous segment is calculated to obtain the audio energy change rate, which is used to measure the audio change.

[0102] Step 7: Two-dimensional dynamic frame adjustment. Dynamic frame adjustment is performed based on the obtained video pixel change rate and audio energy change rate: when both are >0.6 (intense emotional fluctuations), the initial 1-second frame is subdivided into 0.5 seconds / frame; when both are <0.2 (calm emotions), two consecutive 1-second frames are merged into 2 seconds / frame (the first second accounts for 40%, the second for 60%); when between 0.2 and 0.6, the 1-second / frame baseline is maintained. Simultaneously, time deviation is corrected (<0.1 seconds) through cross-modal verification to ensure frame sequence synchronization.

[0103] Step 8: Data preprocessing. Video frames are adjusted to 1280×720 resolution; audio is denoised using wavelet thresholding (threshold 0.02) and L2 normalized; physiological data is segmented according to video frame granularity and time-calibrated (error ≤ 0.05 seconds) to achieve millisecond-level alignment of the three types of data.

[0104] Step 9: Micro-expression feature extraction. Based on the Dilb library, 68 facial key points are extracted from video frames. The displacement and velocity are calculated, and 70-dimensional dynamic features are obtained by Z-score normalization and PCA dimensionality reduction. Combined with the FACS system to label 8 groups of facial muscle contraction intensities (0-100 points), 30-dimensional intensity features are generated and merged into 100 dimensions / frame.

[0105] Step 10: Physiological Feature Extraction. Six indicators (average temperature, temperature difference, etc.) are extracted from the infrared body temperature data in three regions: the forehead, left and right cheekbones, to obtain temperature features; these are input into a three-layer MLP model to output emotional activation levels, which are then combined into overall features to reflect the physiological changes associated with emotions.

[0106] Step 11: Audio Feature Extraction. First, calculate the STFT (Short Time Fourier Transform) and transform it through a Mel filter bank containing 40 filters. Then, obtain the MFCC (Mel Frequency Cepstral Coefficients) through logarithmic and DCT (Discrete Cosine Transform). Supplement with various acoustic parameters such as spectral centroid, bandwidth, and spectral entropy to finally form a feature sequence.

[0107] Step 12: Micro-expression, physiological, and audio features are input into three independent BiLSTM (Bidirectional Long Short-Term Memory) networks, each containing two 256-dimensional hidden layers (128 dimensions each for forward and backward passes), with a dropout rate of 0.3 between layers. The network outputs the hidden state sequences for each modality: micro-expression. ,physiological Audio ,in This represents the total number of frames.

[0108] Step 13: Construct an age-weighted network. This network is a two-layer fully connected structure with 64 nodes in each hidden layer, using the Tanh activation function. The input is the age information of teenagers aged 12-18, represented by one-hot encoding (5 dimensions, corresponding to the age groups of 12-13, 13-14, 14-15, 15-16, and 17-18 years old). The network is trained on adolescent emotion recognition data, learning the differences in sensitivity to micro-expressions, physiological features, and audio features across different age groups, and outputting weight values ​​for the three features (ranging from 0 to 1), with the sum of the three weights being 1.

[0109] Step 14: Weighted Fusion of Hidden States. The micro-expression hidden state sequence obtained in Step 12 is then processed. Physiological latent state sequence Audio hidden state sequence Each of these weighted hidden state sequences is multiplied by the corresponding weights output by the age-weighted network, and then the three weighted hidden state sequences are summed to obtain a multimodal fusion hidden state sequence that fits the age of the adolescent.

[0110] Step 15: Input Feature Preprocessing. The time steps for the three-modal fusion hidden state are 500, and the feature dimension is 512. Layer normalization is performed on the input features by calculating the mean and variance of the features, transforming them into standardized features with a mean of 0 and a variance of 1.

[0111] Step 16: Causal Attention Calculation. First, generate query, key, and value vectors of dimension 128 using a learnable weight matrix (the correlation matrix is ​​a trainable and optimizable parameter matrix) to balance efficiency and expressive power; then calculate the inter-frame correlation strength to obtain the attention score matrix, apply a causal mask (upper triangular matrix) to make the current frame only focus on the current and previous frames, and finally normalize the score matrix to obtain the attention weight distribution of each frame.

[0112] Step 17: Temporal feature enhancement and output. The vector is weighted and summed with 128 as the attention weight value to obtain the initial fused latent state. The residual connection is retained and added to the original trimodal fused latent state F. After normalization, the influence of event trigger frames and emotion response frames is automatically enhanced to obtain the latent state after temporal causal enhancement.

[0113] Step 18: Emotion Weight Prediction. Input the latent state after temporal causal reinforcement into the classification head containing two fully connected layers. After processing, output an 8-dimensional emotion weight vector (corresponding to 8 emotion categories). Then, use L1 normalization to make the weight sum 1.0 to obtain the proportion of each emotion category.

[0114] Step 19: Output in the format of "emotion type + weight".

[0115] Example 3 Based on the same inventive concept, the present invention also provides a multimodal emotion recognition system for adolescents that integrates dynamic temporal optimization and BiLSTM, for implementing the methods described in the foregoing embodiments. The system includes: a data acquisition module, a model building module, and an emotion recognition module. The data acquisition module is used to acquire a multimodal initial dataset of adolescent emotions to be identified; The model building module is used to build a multimodal emotion recognition model for adolescents based on dynamic temporal optimization and BiLSTM. The emotion recognition module is used to obtain multimodal emotion recognition results for adolescents based on the multimodal initial dataset and the adolescent multimodal emotion recognition model.

[0116] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for multimodal emotion recognition in adolescents integrating dynamic temporal optimization and BiLSTM, characterized in that, The method includes: Obtain a multimodal initial dataset of adolescent emotions to be identified; A multimodal emotion recognition model for adolescents is constructed based on dynamic temporal optimization and BiLSTM. Based on the multimodal initial dataset and the adolescent multimodal emotion recognition model, the adolescent multimodal emotion recognition results were obtained.

2. The method according to claim 1, characterized in that, Methods for obtaining a multimodal initial dataset of adolescent emotions to be identified include: Obtain the initial dataset, which includes a high-resolution facial video stream for extracting micro-expressions, an audio stream as audio data, and facial temperature data acquired by an infrared thermal imager. The initial dataset was labeled to obtain a multimodal initial dataset of adolescent emotions.

3. The method according to claim 2, characterized in that, The adolescent multimodal emotion recognition model includes: a dynamic temporal optimization module, a multimodal feature extraction module, a BiLSTM module, a feature dynamic weighting module, and a temporal causal attention module; The dynamic time series optimization module is used to adaptively adjust the time granularity according to emotional fluctuations to achieve multimodal data alignment; The multimodal feature extraction module is used to extract multimodal features based on aligned multimodal data. The BiLSTM module is used to input the extracted multimodal features into the BiLSTM network to obtain the hidden state sequence of each modality feature. The feature dynamic weight module is used to obtain the dynamic weights of each modal feature and combine them with the hidden state sequence of each modal feature to obtain the multimodal fusion hidden state sequence. The temporal causal attention module is used to perform causal attention calculations on the multimodal fusion hidden state sequence to obtain the temporal causal enhanced hidden state.

4. The method according to claim 3, characterized in that, The dynamic time-series optimization module adaptively adjusts the time granularity based on emotional fluctuations, and the methods for achieving multimodal data alignment include: Construct a timeline with a preset precision, and map the original timestamps of each modality in the initial multimodal dataset onto the timeline; Using the time axis as a reference, the data of each modality is initially framed according to a preset time interval to obtain an initial frame sequence; Based on the initial frame sequence, obtain image differences and audio differences; Based on image and audio differences, the initial frame sequence is dynamically adjusted in two dimensions to obtain the adjusted data frame sequence. The adjusted data frame sequence is preprocessed to achieve multimodal data alignment.

5. The method according to claim 3, characterized in that, The multimodal feature extraction module uses methods based on aligned multimodal data to extract multimodal features, including: Based on the Dlib library, facial key points are extracted from high-definition facial video streams, displacement and velocity are calculated, dynamic features are obtained through standardization and PCA dimensionality reduction, and intensity features are obtained by combining the FACS system to mark the contraction intensity of facial muscles. The dynamic features and intensity features are then combined to obtain micro-expression features. Multiple indicators of a preset area are extracted from facial temperature data to obtain temperature features. Facial temperature data is then input into a 3-layer MLP model to obtain emotional activation. The temperature features and emotional activation are then combined to obtain physiological features. The audio stream is subjected to a short-time Fourier transform and then converted by a Mel filter bank. The Mel frequency cepstral coefficients are obtained by taking the logarithm and discrete cosine transform. Combined with preset acoustic parameters, the audio characteristics are obtained.

6. The method according to claim 3, characterized in that, The BiLSTM module inputs the extracted multimodal features into the BiLSTM network to obtain the hidden state sequence of each modality. Methods for this include: The extracted modal features are input into the corresponding BiLSTM network to obtain the hidden state sequence of each modal feature; The BiLSTM network consists of two LSTM networks, a forward network and a backward network, which capture temporal dependencies from the forward and backward directions of the feature sequence, respectively.

7. The method according to claim 3, characterized in that, The feature dynamic weight module obtains the dynamic weights of each modality feature and combines them with the hidden state sequences of each modality feature to obtain the multimodal fusion hidden state sequence. The method includes: Construct an age-weighted network and obtain the dynamic weights of each modality feature based on the age-weighted network; The hidden state sequences of each modality feature are weighted and summed with the dynamic weights of each modality feature to obtain the multimodal fusion hidden state sequence.

8. The method according to claim 3, characterized in that, The temporal causal attention module performs causal attention computation on multimodal fusion hidden state sequences to obtain temporally causal enhanced hidden states. The methods include: The multimodal fusion hidden state sequence is preprocessed to obtain standardized features; Causal attention calculation is performed on standardized features to obtain attention weights; The attention weights are weighted and summed to obtain the initial fusion hidden state. The initial fusion hidden state and the multimodal fusion hidden state sequence are added together and normalized to obtain the temporal causal reinforcement hidden state.

9. The method according to claim 1, characterized in that, Methods for obtaining adolescent multimodal emotion recognition results based on a multimodal initial dataset and an adolescent multimodal emotion recognition model include: The initial multimodal dataset is input into the adolescent multimodal emotion recognition model to obtain the latent state after temporal causal reinforcement. The latent state, enhanced with temporal causality, is input into a classification head containing two fully connected layers to obtain a sentiment weight vector. L1 normalization was performed on the emotion weight vector to obtain the multimodal emotion recognition results for adolescents.

10. A multimodal emotion recognition system for adolescents integrating dynamic temporal optimization and BiLSTM, the system being used to implement the method described in any one of claims 1-9, characterized in that, The system includes: a data acquisition module, a model building module, and an emotion recognition module; The data acquisition module is used to acquire a multimodal initial dataset of adolescent emotions to be identified; The model building module is used to build a multimodal emotion recognition model for adolescents based on dynamic temporal optimization and BiLSTM. The emotion recognition module is used to obtain multimodal emotion recognition results for adolescents based on the multimodal initial dataset and the adolescent multimodal emotion recognition model.

Images