Emotion recognition method and system based on visual and auditory collaboration

Abstract

The invention relates to the technical field of emotion recognition, and discloses an emotion recognition method and system based on visual and auditory synergy, and the method comprises the steps: obtaining an original video stream and an original audio stream, introducing an explicit uncertainty quantification mechanism for a visual mode and an auditory mode, extracting features, and meanwhile, carrying out the recognition of the visual mode and the auditory mode. An uncertainty sequence that assesses its reliability is generated in parallel. Furthermore, the uncertainty sequence is used as a core guiding signal, and a cross-modal cross attention fusion process is dynamically modulated. The mechanism enables the model to adaptively suppress the contribution of an unreliable mode, and focuses on the interaction of high-quality signals, thereby effectively suppressing noise propagation, realizing robust multi-mode information complementation, and finally improving the accuracy of emotion recognition.

Description

Technical Field

[0001] This application relates to the field of emotion recognition technology, and more specifically, to an emotion recognition method and system based on audiovisual collaboration. Background Technology

[0002] Emotion is at the core of human-computer interaction, and analyzing human outward expressions to achieve emotion recognition is the mainstream approach. Single-modal (such as visual or auditory) recognition is easily affected by environmental interference: vision is affected by lighting, occlusion, and posture, while hearing is easily drowned out by noise. Furthermore, human emotional expression has a visual-auditory synergistic characteristic; single-modal information is often ambiguous and difficult to accurately represent true emotions. Therefore, multimodal emotion recognition integrating visual and auditory elements has become a development trend.

[0003] In existing research on emotion recognition based on audiovisual collaboration, the core challenge lies in how to effectively fuse information from different modalities. Early fusion strategies were relatively simple, such as feature-level concatenation (early fusion) or decision-level fusion (late fusion). Feature-level concatenation directly connects the extracted audiovisual feature vectors. While preserving the original information, it ignores the differences in structure and statistical properties between different modalities, struggles to handle temporal misalignment between modalities, and is easily contaminated by noise from one modality. Decision-level fusion first performs emotion prediction for each individual modality and then weights or votes on the prediction results. While simple and robust, this approach loses the fine-grained dynamic correlation information between modalities in time before fusion, failing to capture the collaborative effect of simultaneous viewing and listening. To address these issues, researchers have introduced more complex fusion mechanisms, such as attention-based fusion methods. Attention mechanisms can dynamically adjust modal weights to improve fusion performance, but most models assume that the input signal is reliable at all times and lack the ability to model time-varying uncertainties within modalities. For example, facial occlusion or sudden noise can cause local visual or auditory information to fail. If the model cannot identify and suppress these unreliable signals, it will still incorrectly fuse them, affecting the accuracy and robustness of recognition. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application propose a method and system for emotion recognition based on audiovisual collaboration. Existing technologies treat all modal features equally, ignoring dynamic changes in signal quality, leading to defects such as low-quality information interfering with fusion.

[0005] According to one aspect of this application, an emotion recognition method based on audiovisual collaboration is provided, comprising: acquiring an original video stream and an original audio stream; performing modality-specific feature and uncertainty encoding on the original video stream to obtain a visual feature sequence and a visual uncertainty sequence; performing modality-specific feature and uncertainty encoding on the original audio stream to obtain an auditory feature sequence and an auditory uncertainty sequence; performing uncertainty-guided cross-attention fusion on the visual feature sequence and auditory feature sequence based on the auditory uncertainty sequence and the visual uncertainty sequence to obtain a fused visual feature sequence and a fused auditory feature sequence; performing temporal information aggregation on the fused visual feature sequence and the fused auditory feature sequence to obtain an audiovisual feature temporal aggregation vector; and performing emotion classification on the audiovisual feature temporal aggregation vector to obtain an emotion probability distribution.

[0006] In one possible implementation, modality-specific feature and uncertainty encoding is performed on the original video stream to obtain a visual feature sequence and a visual uncertainty sequence, including: inputting the original video stream into a visual backbone network to obtain a visual intermediate layer feature matrix; inputting the visual intermediate layer features at each time step in the visual intermediate layer feature matrix into a visual feature head to obtain a visual feature sequence; and inputting the visual intermediate layer features at each time step in the visual intermediate layer feature matrix into a visual uncertainty head to obtain a visual uncertainty sequence.

[0007] In one possible implementation, the visual backbone network is a 3D convolutional neural network, the visual feature head is one or more fully connected layers, the visual uncertainty head is a multilayer perceptron model, and the last layer of the multilayer perceptron model uses the sigmoid activation function.

[0008] In one possible implementation, modality-specific feature and uncertainty encoding is performed on the original audio stream to obtain an auditory feature sequence and an auditory uncertainty sequence, including: inputting the original audio stream into an auditory backbone network to obtain a contextual auditory embedding; inputting the contextual auditory embedding into a parallel auditory feature head path and an auditory uncertainty head to obtain the auditory feature sequence and the auditory uncertainty sequence.

[0009] In one possible implementation, uncertainty-guided cross-attention fusion is performed on visual feature sequences and auditory feature sequences based on auditory uncertainty sequences and visual uncertainty sequences to obtain fused visual feature sequences and fused auditory feature sequences. This includes: constructing a visual query matrix from the visual feature sequences; constructing an auditory key matrix and an auditory value matrix from the auditory feature sequences; calculating a visual-auditory original attention score matrix based on the visual query matrix and the auditory key matrix; applying uncertainty guidance to the visual-auditory original attention score matrix based on the visual uncertainty sequence to obtain a guided visual-auditory original attention weight matrix; and performing residual updates on the visual feature sequences based on the guided visual-auditory original attention weight matrix to obtain the fused visual feature sequences.

[0010] In one possible implementation, the visual-auditory original attention score matrix is ​​guided by uncertainty based on a visual uncertainty sequence to obtain a guided visual-auditory original attention weight matrix. This includes: applying the visual uncertainty sequence to each row vector of the visual-auditory original attention score matrix based on a broadcast mechanism to obtain the guided visual-auditory original attention score matrix; and performing weight normalization on the guided visual-auditory original attention score matrix based on the Softmax activation function to obtain the guided visual-auditory original attention weight matrix.

[0011] In one possible implementation, based on a visual uncertainty sequence, uncertainty guidance is applied to the original visual-auditory attention score matrix to obtain a guided original visual-auditory attention weight matrix. This includes: calculating a joint credibility score sequence using a damped harmonic mean method based on the visual uncertainty sequence and the auditory uncertainty sequence; performing a joint mapping association on the visual uncertainty sequence and the auditory uncertainty sequence based on the joint credibility score sequence to obtain a cross-modal full-time correlation matrix; modifying the original visual-auditory attention score matrix based on the cross-modal full-time correlation matrix to obtain a guided original visual-auditory attention score matrix; and performing probabilistic processing on the guided original visual-auditory attention score matrix to obtain the guided original visual-auditory attention weight matrix.

[0012] In one possible implementation, the visual feature sequence is updated with residuals based on the guided visual-auditory original attention weight matrix to obtain the fused visual feature sequence, including: performing matrix multiplication on the guided visual-auditory original attention weight matrix and the auditory feature sequence to obtain the enhanced visual feature sequence; and performing residual connection and layer normalization on the enhanced visual feature sequence and the visual feature sequence to obtain the fused visual feature sequence.

[0013] In one possible implementation, temporal information aggregation of the fused visual feature sequence and the fused auditory feature sequence to obtain a temporal aggregation vector of visual and auditory features includes: concatenating the fused visual feature sequence and the fused auditory feature sequence to obtain a concatenated feature sequence; and inputting the concatenated feature sequence into a temporal aggregation model based on a bidirectional LSTM model to obtain the temporal aggregation vector of visual and auditory features.

[0014] According to another aspect of this application, an emotion recognition system based on audiovisual collaboration is provided, comprising: a multimodal data acquisition module for acquiring original video streams and original audio streams; a visual feature encoding module for performing modality-specific feature and uncertainty encoding on the original video streams to obtain visual feature sequences and visual uncertainty sequences; an auditory feature encoding module for performing modality-specific feature and uncertainty encoding on the original audio streams to obtain auditory feature sequences and auditory uncertainty sequences; a multimodal cross-attention fusion module for performing uncertainty-guided cross-attention fusion on the visual feature sequences and auditory feature sequences based on the auditory uncertainty sequences and visual uncertainty sequences to obtain fused visual feature sequences and fused auditory feature sequences; a temporal information aggregation module for performing temporal information aggregation on the fused visual feature sequences and fused auditory feature sequences to obtain audiovisual feature temporal aggregation vectors; and an emotion classification module for performing emotion classification on the audiovisual feature temporal aggregation vectors to obtain an emotion probability distribution.

[0015] Compared to existing technologies, the emotion recognition method and system based on audiovisual collaboration provided in this application acquires raw video and audio streams and introduces an explicit uncertainty quantification mechanism for both visual and auditory modalities. While extracting features, it simultaneously generates uncertainty sequences to evaluate their reliability. Furthermore, this uncertainty sequence serves as the core guiding signal to dynamically modulate the cross-modal attention fusion process. This mechanism enables the model to adaptively suppress contributions from unreliable modalities, focusing on the interaction of high-quality signals, thereby effectively suppressing noise propagation, achieving robust multimodal information complementarity, and ultimately improving the accuracy of emotion recognition. Attached Figure Description

[0016] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1The illustration shows a schematic flowchart of an emotion recognition method based on audiovisual collaboration according to an embodiment of this application.

[0018] Figure 2 The illustration shows a schematic flowchart of step S2 in the audiovisual-coordinated emotion recognition method according to an embodiment of this application.

[0019] Figure 3 The illustration shows a schematic flowchart of step S3 in the emotion recognition method based on audiovisual collaboration according to an embodiment of this application.

[0020] Figure 4 The illustration shows a schematic flowchart of step S4 in the audiovisual-coordinated emotion recognition method according to an embodiment of this application.

[0021] Figure 5 The illustration shows a schematic flowchart of step S5 in the emotion recognition method based on audiovisual collaboration according to an embodiment of this application.

[0022] Figure 6 The illustration shows a schematic block diagram of an emotion recognition system based on audiovisual collaboration according to an embodiment of this application. Detailed Implementation

[0023] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0024] Figure 1 The illustration shows a schematic flowchart of an emotion recognition method and system based on audiovisual collaboration according to an embodiment of this application. Figure 1 As shown, this application provides an emotion recognition method based on audiovisual collaboration, including: S1, acquiring the original video stream and the original audio stream; S2, performing modality-specific feature and uncertainty encoding on the original video stream to obtain a visual feature sequence and a visual uncertainty sequence; S3, performing modality-specific feature and uncertainty encoding on the original audio stream to obtain an auditory feature sequence and an auditory uncertainty sequence; S4, performing uncertainty-guided cross-attention fusion on the visual feature sequence and the auditory feature sequence based on the auditory uncertainty sequence and the visual uncertainty sequence to obtain a fused visual feature sequence and a fused auditory feature sequence; S5, performing temporal information aggregation on the fused visual feature sequence and the fused auditory feature sequence to obtain an audiovisual feature temporal aggregation vector; S6, performing emotion classification on the audiovisual feature temporal aggregation vector to obtain an emotion probability distribution.

[0025] For example, in step S1, the original video stream and the original audio stream are acquired. It should be understood that human emotional expression is essentially a complex, multimodal, and collaborative behavior. The inner emotional state is not displayed in isolation through a single channel, but rather synchronously and complementaryly manifested in both visual and auditory dimensions. Visual information, such as subtle twitches of facial muscles, changes in eye contact, and the curvature of the corners of the mouth, constitutes the intuitive carrier of emotional expression; while auditory information, such as the pitch, volume, speed, and timbre of speech, conveys the intensity and inherent rhythm of emotion. The limitations of relying solely on a single modality for analysis are obvious. For example, pure visual analysis is easily affected by factors such as lighting conditions, partial facial occlusion (e.g., hands, masks), or head posture, leading to the loss or distortion of key feature information. Similarly, pure auditory analysis is also difficult to escape the influence of environmental factors such as background noise, multi-person conversations, or channel distortion, causing effective emotional signals to be submerged. More importantly, unimodal information is often ambiguous. A static smile, when paired with a light, upbeat tone, can be interpreted as joy; however, when paired with a flat, somber tone, it may simply indicate politeness or helplessness. Therefore, to overcome the inherent limitations and external interference of unimodal information and achieve a comprehensive, accurate, and unambiguous understanding of emotional states, it is essential to simultaneously capture and analyze these two interconnected and mutually reinforcing information channels.

[0026] Specifically, the acquisition of raw video and audio streams can be achieved in various ways, including loading from publicly available, pre-labeled sentiment databases or real-time acquisition via hardware devices. After acquiring the raw data, the video and audio streams need to be processed separately. For video streams, the processing includes: first, decoding the video file into a series of independent image frames; then, to ensure both temporal integrity and computational efficiency, sampling the image frame sequence at a fixed frame rate; subsequently, since the face is the core area for emotional expression, face detection algorithms can be used to locate and crop the face region in each frame to eliminate interference from irrelevant background information; finally, to meet the uniform input size requirements of subsequent neural network models, the cropped face images are scaled to a fixed resolution. For audio streams, the processing includes: first, resampling the original audio, unifying its sampling rate to a standard value to ensure data consistency; second, typically converting it to mono audio to simplify subsequent feature extraction. Through this series of preprocessing operations, the raw, unstructured audiovisual data is transformed into a data sequence with a uniform format and standardized size, which can be directly input into subsequent models.

[0027] To illustrate this in more detail, a specific embodiment is provided. In this embodiment, firstly, raw video and audio streams containing human emotional expressions are acquired. This data can originate from publicly available emotion datasets, such as CREMA-D and MSP-IMPROV, or can be acquired in real-time using an integrated camera and microphone. For the acquired raw video stream, the system decodes it into a continuous sequence of image frames and samples it at a rate of 25 frames per second (fps). Subsequently, for each frame in the sequence, the system uses advanced face detection algorithms such as MTCNN (Multi-task Cascaded Convolutional Networks) to accurately locate and crop the face region, and then uniformly scales all cropped face images to 224x224 pixels. Simultaneously, for the acquired raw audio stream, the system uniformly resamples it to a sampling rate of 16kHz and performs mono conversion. Finally, the standardized video frame sequence and audio waveform data are output.

[0028] For example, in step S2, modality-specific feature and uncertainty encoding is performed on the original video stream to obtain a visual feature sequence and a visual uncertainty sequence. It should be understood that traditional feature extraction methods typically focus only on extracting high-dimensional features that can characterize the core content from the data, but they implicitly assume a key assumption: that the input signal at all times has equal quality and reliability. However, in real-world applications, this assumption often fails. Visual signals are highly susceptible to interference from various uncontrollable factors. For example, drastic changes in lighting conditions may cause facial features to be overexposed or underexposed; unconscious hand movements or wearing masks can obscure key facial areas (such as the mouth and nose); and extreme head posture deviations may prevent facial features from being fully captured. These factors introduce noise and cause information loss within a specific time period, significantly reducing the quality of visual features during that period. If subsequent fusion modules indiscriminately fuse these low-quality or even invalid features with high-quality auditory features, not only will effective modal complementarity fail, but the introduction of noise will also contaminate the overall feature representation, ultimately leading to a significant decrease in the accuracy and robustness of emotion recognition. Therefore, this application aims to transcend traditional feature extraction paradigms, not only to answer the question "what was seen," but also to simultaneously and quantitatively answer the question "how clearly was seen." By encoding information in parallel across two dimensions—content (features) and quality (uncertainty)—it provides indispensable and refined guidance for subsequent intelligent fusion decisions.

[0029] In one embodiment, such as Figure 2As shown, modality-specific feature and uncertainty encoding is performed on the original video stream to obtain a visual feature sequence and a visual uncertainty sequence, including: S21, inputting the original video stream into a visual backbone network to obtain a visual intermediate layer feature matrix; S22, inputting the visual intermediate layer features at each time step in the visual intermediate layer feature matrix into a visual feature head to obtain a visual feature sequence; S23, inputting the visual intermediate layer features at each time step in the visual intermediate layer feature matrix into a visual uncertainty head to obtain a visual uncertainty sequence.

[0030] Specifically, firstly, the original video stream is input as a whole into a visual backbone network. This backbone network is preferably a 3D Convolutional Neural Network (3D-CNN), whose three-dimensional convolutional kernel structure can simultaneously operate on both the spatial dimension (width and height) of the image and the temporal dimension of the sequence, thus naturally capturing the complete spatiotemporal dynamic information of facial expressions from occurrence to end, rather than merely a series of isolated static snapshots. After multi-level feature extraction and abstraction, the backbone network outputs a high-dimensional visual intermediate layer feature matrix. This matrix is ​​aligned with the input video frame sequence in the temporal dimension and contains rich spatiotemporal contextual information at each time point. Subsequently, this visual intermediate layer feature matrix is ​​segmented in the temporal dimension, and the intermediate layer features corresponding to each time step are input in parallel into two functionally distinct but structurally independent head networks. One is the visual feature head, which consists of one or more fully connected layers. Its responsibility is to map and refine the high-dimensional intermediate layer features into a feature vector of appropriate dimension. The outputs of all time steps constitute the final visual feature sequence. The second is the visual uncertainty head, which employs a multilayer perceptron (MLP) model, with the last layer specifically using the sigmoid activation function. The sigmoid function maps its input values ​​to a range of 0 to 1, giving its output a natural physical meaning of probability or confidence. Therefore, this uncertainty head maps the intermediate layer features at each time step to a scalar value, which (e.g., close to 1 indicates high reliability, close to 0 indicates high unreliability) directly quantifies the quality of the visual information at that moment. The scalar values ​​output from all time steps constitute a visual uncertainty sequence that corresponds one-to-one with the visual feature sequence in time. Through this implementation, the present invention successfully achieves synchronous and decoupled encoding of visual information content and quality within a single architecture, laying a solid foundation for subsequent steps.

[0031] For example, in step S3, modality-specific feature and uncertainty encoding is performed on the original audio stream to obtain an auditory feature sequence and an auditory uncertainty sequence. It should be understood that, similar to visual signals, auditory signals also face the severe challenge of unstable quality during actual acquisition and transmission. For example, in open environments, persistent or transient background noise (such as traffic noise or office noise) can overlap with the target speech signal; in multi-person dialogue scenarios, interference from non-target voices can severely affect the extraction of emotional information from the main speaker; furthermore, differences in the performance of acquisition devices such as microphones, distortion of communication channels, and unclear pronunciation by the speaker (such as whispering or coughing) can all lead to significant fluctuations in the signal-to-noise ratio and information completeness of the audio stream at different time segments. If traditional methods are used, extracting only a single feature sequence from the audio stream makes it impossible to distinguish which features at which time points are clear and reliable, and which are severely contaminated by noise. This could lead to low-quality auditory features being incorrectly assigned excessive weights in subsequent modality fusion stages. This would not only fail to effectively supplement visual information but would also introduce interference, lowering the overall performance ceiling of the multimodal system. Therefore, this application argues that a mechanism capable of simultaneously evaluating the content and quality of auditory information must be established. This is achieved by encoding an auditory feature sequence representing emotional content and an auditory uncertainty sequence representing signal reliability in parallel.

[0032] In one embodiment, such as Figure 3 As shown, modality-specific feature and uncertainty encoding is performed on the original audio stream to obtain an auditory feature sequence and an auditory uncertainty sequence, including: S31, inputting the original audio stream into an auditory backbone network to obtain a contextual auditory embedding; S32, inputting the contextual auditory embedding into a parallel auditory feature head path and an auditory uncertainty head to obtain the auditory feature sequence and the auditory uncertainty sequence.

[0033] Specifically, firstly, the raw audio stream (typically one-dimensional waveform data) is directly input into an auditory backbone network. This backbone network is preferably an advanced, self-supervised learning-based pre-trained model (e.g., Wav2Vec 2.0). Such models can directly learn powerful acoustic representations from massive amounts of unlabeled audio data, effectively capturing rich context from low-level acoustic features to high-level semantic information. Its output is a high-dimensional contextual auditory embedding matrix. This matrix is ​​aligned with the input audio stream in the temporal dimension and encodes complex auditory information at each time point. Subsequently, this step enters its core parallel processing stage: the contextual auditory embedding matrix is ​​simultaneously input into two parallel processing pathways. The first is the auditory feature head pathway, which typically consists of one or more fully connected layers. Its function is to map the high-dimensional contextual auditory embedding into a pre-defined feature space that matches the visual feature dimensions, thereby generating the final auditory feature sequence. The second is the auditory uncertainty head, which also receives the contextual auditory embedding and processes it through an independent multilayer perceptron (MLP) model. Consistent with the processing approach for the visual modality, the final layer of this MLP also employs the sigmoid activation function, thus mapping the auditory features at each time step to a scalar value between 0 and 1. This scalar value directly quantifies the reliability of the auditory information at the corresponding moment, and the outputs of all time steps constitute a temporally corresponding sequence of auditory uncertainty.

[0034] For example, in step S4, the visual feature sequence and the auditory feature sequence are fused using uncertainty-guided cross-attention based on the auditory uncertainty sequence and the visual uncertainty sequence to obtain the fused visual feature sequence and the fused auditory feature sequence. It should be understood that traditional multimodal fusion techniques, whether early feature layer splicing or later attention-based fusion, often implicitly assume that all input features have equal validity and reliability at any given time. However, in real-world applications, this idealized assumption is seriously inconsistent with reality. The quality of audiovisual signals is dynamic. For example, visual modalities may experience severe distortion or loss of feature information within a specific time period due to factors such as sudden changes in illumination, facial occlusion by objects or limbs, or rapid head rotation. Simultaneously, auditory modalities may also be affected by background noise, crosstalk from multiple conversations, packet loss or distortion during signal transmission, etc.

[0035] Traditional fusion mechanisms lack the ability to perceive and adapt to dynamic changes in signal quality. When a feature of one modality becomes unreliable due to interference at a certain point in time, a blind fusion model will still treat it as valid information and participate in cross-modal interactive computation. This behavior not only fails to achieve complementary advantages between modalities but also leads to the negative effect of noise propagation. That is, noise or invalid information from one modality will contaminate another originally clear modality, thereby lowering the overall quality of feature representation and ultimately severely damaging the accuracy and generalization ability of the emotion recognition system in complex environments. Therefore, this application argues that the core of an advanced multimodal fusion mechanism should not merely be the simple aggregation of information but must be an intelligent, selective, and adaptive interactive process. The fundamental purpose of this step is to establish such an intelligent fusion framework. It no longer passively accepts all features but actively utilizes the visual uncertainty sequence and auditory uncertainty sequence generated in the preceding steps, which correspond one-to-one with the feature sequence in time, as dynamic and quantifiable guiding signals. By combining signal characteristics with uncertainty, the model can evaluate the reliability of each information fragment in real time and with fine precision during the fusion process. This allows for dynamic adjustment of the interaction strength between modalities, suppression of the propagation of unreliable information, and amplification of the guiding role of reliable information. This is the only way to achieve truly robust multimodal collaborative understanding.

[0036] In one embodiment, such as Figure 4 As shown, uncertainty-guided cross-attention fusion is performed on visual feature sequences and auditory feature sequences based on auditory uncertainty sequences and visual uncertainty sequences to obtain fused visual feature sequences and fused auditory feature sequences. The process includes: S41, constructing a visual query matrix from the visual feature sequences; S42, constructing an auditory key matrix and an auditory value matrix from the auditory feature sequences; S43, calculating the original visual-auditory attention score matrix based on the visual query matrix and the auditory key matrix; S44, applying uncertainty guidance to the original visual-auditory attention score matrix based on the visual uncertainty sequence to obtain a guided original visual-auditory attention weight matrix; and S45, performing residual updates on the visual feature sequences based on the guided original visual-auditory attention weight matrix to obtain the fused visual feature sequence.

[0037] Specifically, firstly, the query, key, and value matrices are constructed. This involves using a linear transformation to construct a visual query matrix from the visual feature sequence, and simultaneously using two independent linear transformations to construct an auditory key matrix and an auditory value matrix from the auditory feature sequence. Secondly, the raw attention score is calculated by performing matrix multiplication on the visual query matrix and the auditory key matrix to obtain a visual-auditory raw attention score matrix. This matrix initially quantifies the degree of attention given to each auditory time step by each visual time step.

[0038] Next, based on the visual uncertainty sequence, uncertainty guidance is applied to the original visual-auditory attention score matrix to obtain the guided original visual-auditory attention weight matrix. In one embodiment, applying uncertainty guidance to the original visual-auditory attention score matrix based on the visual uncertainty sequence to obtain the guided original visual-auditory attention weight matrix includes: First, applying the visual uncertainty sequence to each row vector of the original visual-auditory attention score matrix using a broadcast mechanism to obtain the guided original visual-auditory attention score matrix. Specifically, each row of the original visual-auditory attention score matrix is ​​multiplied by its corresponding visual uncertainty value. This means that if a feature at a certain visual time step is unreliable (its uncertainty value is low), its willingness to focus on all auditory features will be suppressed overall. Second, the guided original visual-auditory attention score matrix is ​​normalized using a Softmax activation function to obtain the guided original visual-auditory attention weight matrix.

[0039] Furthermore, regarding visual and auditory uncertainty sequences, considering that intermodal uncertainties may also be strongly correlated over time, their synergistic effects contain richer information than their independent effects. Specifically, intermodal uncertainties may exhibit a synergistic negative correlation in the temporal dimension. For example, when a user covers their mouth with their hand while speaking, visual uncertainty (mouth area) increases, and auditory uncertainty (dull, unclear sound) also increases. Therefore, the combined effect of these two simultaneous uncertainties should produce a stronger inhibitory effect than any single event; that is, the model should know that the pairing of visual and auditory signals at this moment is highly unreliable. In addition, intermodal uncertainties may also have compensatory relationships. For example, when a user's facial expressions are exaggerated (low visual uncertainty) but they are in a noisy environment (high auditory uncertainty), the model should trust the visual-to-auditory guidance more, and vice versa.

[0040] Therefore, in another embodiment, uncertainty guidance is applied to the original visual-auditory attention score matrix based on the visual uncertainty sequence to obtain the guided original visual-auditory attention weight matrix. This includes: calculating a joint credibility score sequence using a damped harmonic mean method based on the visual uncertainty sequence and the auditory uncertainty sequence; performing a joint mapping association on the visual uncertainty sequence and the auditory uncertainty sequence based on the joint credibility score sequence to obtain a cross-modal full-time correlation matrix; modifying the original visual-auditory attention score matrix based on the cross-modal full-time correlation matrix to obtain the guided original visual-auditory attention score matrix; and performing probabilistic processing on the guided original visual-auditory attention score matrix to obtain the guided original visual-auditory attention weight matrix.

[0041] Specifically, for visual uncertainty sequences, for example, represented as And auditory uncertainty sequences, for example, represented as The joint confidence score along the time-series dimension is output based on the damped harmonic mean method, and is expressed as: ;in, Represents the first in the visual uncertainty sequence 1 eigenvalue, Represents the first in the auditory uncertainty sequence 1 eigenvalue, Indicates the first A joint credibility score, It is a small positive number (e.g., 10). -6 This is to prevent numerical inflation introduced by a denominator of zero (e.g., where both visual and auditory features are completely unreliable) or a very small denominator. Here, the harmonic mean is relatively sensitive to small values, reflecting the bottleneck effect or logic, i.e., only the corresponding... and When both are relatively high, Only then will the reliability be relatively high, thus reflecting the joint reliability. In other words, it embodies a serial reliability model, assuming that the reliability of joint interactions is limited by the least reliable link. and If any one of them approaches 0, then even if the other is 1, It will also approach 0, and it also shows that if both visual and auditory features are unreliable, the joint reliability should drop sharply.

[0042] Then, for the joint credibility matrix composed of joint credibility scores For visual uncertainty sequences in column vector form, respectively and auditory uncertainty sequence Perform a joint mapping association, that is: ;in, Represents matrix multiplication. Represents a visual uncertainty sequence. This represents an auditory uncertainty sequence. and It is a column vector. Indicates the transpose symbol. This represents the joint credibility matrix, thus obtaining a cross-modal, full-time correlation matrix representing the uncertainty of the joint credibility based on visual and auditory features. .

[0043] In this way, the cross-modal full-time correlation matrix can be used. Let the original visual-auditory attention score matrix be denoted as follows: Corrections were made. Additionally, the cross-modal full-time correlation matrix was taken into account. Relative to the matrix dimensional symmetry of the joint reliable representation of uncertainties in the visual and auditory dimensions, the original visual-auditory attention score matrix is ​​modified based on the cross-modal full-time correlation matrix: ;in, Represents the cross-modal full-time correlation matrix. This represents the original visual-auditory attention score matrix. Represents matrix addition. This represents the original visual-auditory attention score matrix after guidance.

[0044] Finally, the guided visual-auditory raw attention score matrix was analyzed. Apply probabilistic functions (e.g.) The function is used to process the data to obtain the original visual-auditory attention weight matrix after guidance. Thus, through the aforementioned collaborative uncertainty modulation cross-attention mechanism, improved joint noise and occlusion suppression capabilities can be achieved, for example, in the case of speaking while covering one's mouth. and The values ​​are very low at the corresponding time steps, thus the values ​​calculated using the harmonic mean are... The value will also become extremely low, far lower than that of a single [value]. and This improved representation allows the model to almost completely ignore these invalid cross-modal interactions, effectively preventing the propagation and fusion of erroneous information and significantly enhancing robustness. Furthermore, it enables more refined cross-modal information filtering, such as assuming visual clarity (all... (All are high), but the noise is only present in certain segments (partial). (low), then The matrix will only display low values ​​in the corresponding columns, meaning that even within the same visual timestep, interactions with other auditory timesteps will be differentiated based on the context of those interactions. This allows the model to learn to dynamically and selectively extract information from the effective portions of the auditory signal to enhance visual understanding. Furthermore, by strongly penalizing unreliable joint pairs, the model needs to find time-aligned regions where both visual and auditory signals are clear and reliable during training. This effectively adds strong prior knowledge to the attention mechanism, helping the model learn meaningful audiovisual causal relationships and correlations more quickly and accurately, rather than just statistical co-occurrence, thus promoting effective cross-modal alignment learning.

[0045] Finally, based on the guided attention weight matrix, the visual feature sequence is updated using residuals to obtain the fused visual feature sequence. This process specifically includes: performing matrix multiplication between the guided visual-auditory original attention weight matrix and the auditory value matrix to obtain an enhanced visual feature sequence selectively aggregated from the auditory modality; then, performing residual connection and layer normalization on this enhanced visual feature sequence and the original visual feature sequence to obtain the final fused visual feature sequence. Symmetrically, using the exact same mechanism, with auditory features as the query and visual features as the information provider, and utilizing the auditory uncertainty sequence for guidance, a fused auditory feature sequence can be obtained.

[0046] For example, in step S5, the fused visual feature sequence and the fused auditory feature sequence are aggregated for temporal information to obtain a temporal aggregated vector of visual and auditory features. It should be understood that the fused visual feature sequence and the fused auditory feature sequence are essentially time-series data; that is, each of their feature vectors corresponds to a specific point in time in the input audiovisual segment, collectively depicting the dynamic trajectory of emotional state evolution over time. However, the ultimate goal of the emotion classification task is to provide a single, global emotion label for the entire audiovisual segment. This requires that the input to the classifier must be a fixed-dimensional feature vector that can summarize and represent all information within the entire time period, rather than a feature sequence of variable length.

[0047] Based on this, time-series information aggregation of the fused visual feature sequence and the fused auditory feature sequence effectively and losslessly compresses and transforms this dynamic, temporal, high-dimensional feature stream into a static, global, fixed-dimensional single vector representation. In one embodiment, average pooling or max pooling can be performed on all feature vectors in the sequence along the time dimension. However, while this achieves dimensionality reduction and fixed length, it comes at the cost of completely sacrificing the temporal structural information in the sequence. Human emotional expression is a dynamic behavior with inherent logic and an evolutionary process, and the correlation between its preceding and following information is crucial. For example, a sequence of facial expressions that first frowns and then smiles carries a completely different emotional meaning than a sequence that first smiles and then frowns. Simple pooling operations cannot distinguish between these two cases because they treat every point in time equally, ignoring the order and dependencies between them. This neglect of temporal information leads to a lack of understanding of the dynamic evolution of emotions, thus severely limiting the performance ceiling that the model can achieve. Therefore, this application argues that a technical means capable of deeply understanding and modeling long-range temporal dependencies must be adopted to perform this aggregation process. The purpose of this step is to use a powerful temporal aggregation model to highly condense the rich dynamic information, contextual dependencies, and overall evolution trends contained in the fused audiovisual feature sequence into a single, information-dense audiovisual feature temporal aggregation vector, providing the highest quality and most comprehensive feature input for subsequent accurate classification.

[0048] In one embodiment, such as Figure 5 As shown, the temporal information aggregation of the fused visual feature sequence and the fused auditory feature sequence to obtain the audiovisual feature temporal aggregation vector includes: S51, concatenating the fused visual feature sequence and the fused auditory feature sequence to obtain the concatenated feature sequence; S52, inputting the concatenated feature sequence into a temporal aggregation model based on a bidirectional LSTM model to obtain the audiovisual feature temporal aggregation vector.

[0049] Specifically, firstly, the fused visual and auditory feature sequences are concatenated at each corresponding time step. Specifically, at any time step t (t ranges from 1 to T), the fused visual feature vector and the fused auditory feature vector at that time are concatenated to form a concatenated feature vector. Combining the concatenation results from all time steps yields a concatenated feature sequence. This tightly integrates the enhanced information from the two modalities at each time step, providing a single, comprehensive input data stream for subsequent temporal modeling.

[0050] Next, the concatenated feature sequence is input into a temporal aggregation model based on a bidirectional Long Short-Term Memory (Bi-LSTM) model. The Bi-LSTM model is chosen for its powerful temporal modeling capabilities. On one hand, the gating mechanisms within the LSTM unit, such as forget gates, input gates, and output gates, enable it to effectively learn and memorize long-range dependencies in the sequence, overcoming the gradient vanishing or exploding problems present in traditional recurrent neural networks (RNNs). On the other hand, its bidirectional structure is crucial for capturing complete emotional context. A standard, unidirectional LSTM can only process the current time step based on past information, while emotional understanding often relies on global context. For example, the true emotion of a flat tone at the beginning of a speech may need to be accurately judged by combining it with the rising tone at the end. Bi-LSTM processes the input sequence simultaneously from the beginning to the end and from the end to the beginning by setting up a forward LSTM and a backward LSTM. At any given time t, the hidden state of the output contains context information from before time t (from the forward LSTM) and after time t (from the backward LSTM).

[0051] After the Bi-LSTM model processes the entire concatenated feature sequence, it is necessary to extract the final single vector representing global information. Specifically, this is achieved by extracting the hidden state vectors of the forward LSTM at the last time step and the backward LSTM at the first time step. The former summarizes all information from the beginning to the end of the sequence, while the latter summarizes all information from the end to the beginning. Concatenating these two hidden state vectors yields the final, fixed-length temporal aggregation vector of the audiovisual features. This vector highly condenses all the temporal dynamics and deep contextual information of the multimodal features after robust fusion throughout the entire audiovisual segment.

[0052] For example, in step S6, the temporal aggregation vector of audiovisual features is subjected to sentiment classification to obtain a sentiment probability distribution. It should be understood that this temporal aggregation vector of audiovisual features is the final product obtained after in-depth analysis, denoising, fusion, and condensation of the input audiovisual segments. Mathematically, it is a point in a high-dimensional space, and its location contains all the key information about the emotional state. However, this vector itself does not directly correspond to any specific sentiment category; it is an intermediate, potential representation. Without sentiment classification, the entire technical solution would stop at a numerical vector that cannot be directly interpreted, failing to fulfill its core function as a sentiment recognition system. Therefore, by constructing a discriminative model as a bridge connecting the feature space and the semantic space, the temporal aggregation vector of audiovisual features is finally interpreted and judged, transforming it into a clear, quantifiable sentiment classification result.

[0053] Furthermore, considering the inherent complexity and ambiguity of human emotional expression, certain facial expressions or tones may simultaneously contain multiple emotional components, or their intensity may vary. If the system only outputs a hard label (e.g., "happy"), it will lose a large amount of valuable subtle information and fail to reflect the model's confidence in the decision. An emotional probability distribution, essentially a vector, where each element corresponds to the probability of a preset emotional category (e.g., happy, sad, angry, neutral, etc.), and the sum of all elements is one. This form of output is superior because it not only provides the most likely emotional category (i.e., the category with the highest probability) but also offers the model's assessment of other possible emotional categories. This provides richer decision-making support for subsequent application systems and better reflects the uncertain nature of emotional expression in the real world.

[0054] In one embodiment, an audiovisual feature temporal aggregation vector is input into a classifier module consisting of one or more fully connected neural network layers and a final activation function to perform sentiment classification and obtain a sentiment probability distribution. Specifically, firstly, the audiovisual feature temporal aggregation vector is passed as input to the classifier. The core of the classifier is a fully connected layer (also called a linear layer). The number of input neurons in this fully connected layer is set to be exactly the same as the dimension of the audiovisual feature temporal aggregation vector to ensure that complete feature information can be received. The number of its output neurons is precisely set to be equal to the total number K of the predefined sentiment categories to be identified in this application. For example, if the task is to identify six basic sentiments, then the number of output neurons in this layer is 6. This fully connected layer performs a linear transformation on the input aggregation vector through a learnable weight matrix and a bias vector, calculating a raw, unnormalized score for each sentiment category.

[0055] After obtaining this set of raw scores, they are input into the Softmax activation function. The Softmax function is a special non-linear function specifically designed to transform a vector containing arbitrary real values ​​into a valid probability distribution. Its specific calculation method is as follows: for the ... Raw scores for each sentiment category Its final probability The result is obtained through the following formula: ;in, Indicates the first The raw scores for each sentiment category, Indicates the first The raw scores for each sentiment category, This represents the total number of predefined emotion categories. Indicates the first The formula calculates the probability of each sentiment category. First, it maps all the original scores to the positive domain using an exponential function. Then, it normalizes the output by dividing the exponent of each score by the sum of the exponents of all scores. After processing with the Softmax function, each element of the output vector is strictly between 0 and 1, and the sum of all elements is exactly equal to 1. This final output vector is the sentiment probability distribution.

[0056] In summary, the emotion recognition method based on audiovisual collaboration provided in this application acquires the original video and audio streams and introduces an explicit uncertainty quantification mechanism for both the visual and auditory modalities. While extracting features, it simultaneously generates uncertainty sequences to evaluate their reliability. Furthermore, this uncertainty sequence serves as the core guiding signal to dynamically modulate the cross-modal attention fusion process. This mechanism enables the model to adaptively suppress contributions from unreliable modalities, focusing on the interaction of high-quality signals, thereby effectively suppressing noise propagation, achieving robust multimodal information complementarity, and ultimately improving the accuracy of emotion recognition.

[0057] This application also provides an emotion recognition system based on audiovisual collaboration, used to perform the aforementioned emotion recognition method based on audiovisual collaboration, such as... Figure 6As shown, the audiovisual-based emotion recognition system 600 includes: a multimodal data acquisition module 610 for acquiring original video streams and original audio streams; a visual feature encoding module 620 for performing modality-specific feature and uncertainty encoding on the original video stream to obtain a visual feature sequence and a visual uncertainty sequence; an auditory feature encoding module 630 for performing modality-specific feature and uncertainty encoding on the original audio stream to obtain an auditory feature sequence and an auditory uncertainty sequence; a multimodal cross-attention fusion module 640 for performing uncertainty-guided cross-attention fusion on the visual feature sequence and auditory feature sequence based on the auditory uncertainty sequence and visual uncertainty sequence to obtain a fused visual feature sequence and a fused auditory feature sequence; a temporal information aggregation module 650 for performing temporal information aggregation on the fused visual feature sequence and the fused auditory feature sequence to obtain an audiovisual feature temporal aggregation vector; and an emotion classification module 660 for performing emotion classification on the audiovisual feature temporal aggregation vector to obtain an emotion probability distribution.

[0058] This application also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the emotion recognition method based on audiovisual collaboration provided in the above embodiments.

[0059] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to achieve the emotion recognition method based on audiovisual collaboration provided in the above embodiments.

[0060] In this application, the system, computer-readable storage medium, or computer program product provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0061] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments.

[0062] The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. An emotion recognition method based on visual-auditory coordination, characterized in that, include: Acquire the raw video stream and raw audio stream; perform modality-specific feature and uncertainty encoding on the raw video stream to obtain visual feature sequence and visual uncertainty sequence; Modality-specific features and uncertainty encoding are performed on the original audio stream to obtain auditory feature sequences and auditory uncertainty sequences; based on the auditory uncertainty sequences and visual uncertainty sequences, uncertainty-guided cross-attention fusion is performed on the visual feature sequences and auditory feature sequences to obtain fused visual feature sequences and fused auditory feature sequences; Temporal information aggregation is performed on the fused visual feature sequence and the fused auditory feature sequence to obtain a temporal aggregated vector of visual and auditory features; sentiment classification is performed on the temporal aggregated vector of visual and auditory features to obtain the sentiment probability distribution.

2. The emotion recognition method based on audiovisual collaboration according to claim 1, characterized in that, Modality-specific feature and uncertainty encoding is performed on the original video stream to obtain a visual feature sequence and a visual uncertainty sequence, including: inputting the original video stream into a visual backbone network to obtain a visual intermediate layer feature matrix; inputting the visual intermediate layer features at each time step in the visual intermediate layer feature matrix into a visual feature head to obtain a visual feature sequence; and inputting the visual intermediate layer features at each time step in the visual intermediate layer feature matrix into a visual uncertainty head to obtain a visual uncertainty sequence.

3. The emotion recognition method based on audiovisual collaboration according to claim 2, characterized in that, The visual backbone network is a 3D convolutional neural network, the visual feature head is one or more fully connected layers, the visual uncertainty head is a multilayer perceptron model, and the last layer of the multilayer perceptron model uses the sigmoid activation function.

4. The emotion recognition method based on audiovisual collaboration according to claim 1, characterized in that, Modality-specific feature and uncertainty encoding is performed on the original audio stream to obtain an auditory feature sequence and an auditory uncertainty sequence, including: inputting the original audio stream into an auditory backbone network to obtain a contextual auditory embedding; inputting the contextual auditory embedding into a parallel auditory feature head path and an auditory uncertainty head to obtain the auditory feature sequence and the auditory uncertainty sequence.

5. The emotion recognition method based on audiovisual collaboration according to claim 1, characterized in that, Based on auditory and visual uncertainty sequences, uncertainty-guided cross-attention fusion is performed on visual and auditory feature sequences to obtain fused visual and auditory feature sequences. This includes: constructing a visual query matrix from the visual feature sequences; constructing an auditory key matrix and an auditory value matrix from the auditory feature sequences; calculating a visual-auditory original attention score matrix based on the visual query matrix and the auditory key matrix; applying uncertainty guidance to the visual-auditory original attention score matrix based on the visual uncertainty sequence to obtain a guided visual-auditory original attention weight matrix; and performing residual updates on the visual feature sequences based on the guided visual-auditory original attention weight matrix to obtain the fused visual feature sequence.

6. The emotion recognition method based on audiovisual collaboration according to claim 5, characterized in that, Based on the visual uncertainty sequence, uncertainty guidance is applied to the original visual-auditory attention score matrix to obtain the guided original visual-auditory attention weight matrix. This includes: applying the visual uncertainty sequence to each row vector of the original visual-auditory attention score matrix based on a broadcast mechanism to obtain the guided original visual-auditory attention score matrix; and performing weight normalization on the guided original visual-auditory attention score matrix based on the Softmax activation function to obtain the guided original visual-auditory attention weight matrix.

7. The emotion recognition method based on audiovisual collaboration according to claim 5, characterized in that, Based on the visual uncertainty sequence, uncertainty guidance is applied to the original visual-auditory attention score matrix to obtain the guided original visual-auditory attention weight matrix. This includes: calculating a joint credibility score sequence using a damped harmonic mean method based on the visual and auditory uncertainty sequences; performing a joint mapping association on the visual and auditory uncertainty sequences based on the joint credibility score sequence to obtain a cross-modal full-time correlation matrix; modifying the original visual-auditory attention score matrix based on the cross-modal full-time correlation matrix to obtain the guided original visual-auditory attention score matrix; and performing probabilistic processing on the guided original visual-auditory attention score matrix to obtain the guided original visual-auditory attention weight matrix.

8. The emotion recognition method based on audiovisual collaboration according to claim 5, characterized in that, Based on the guided visual-auditory original attention weight matrix, residual updates are performed on the visual feature sequence to obtain the fused visual feature sequence, including: performing matrix multiplication on the guided visual-auditory original attention weight matrix and the auditory feature sequence to obtain the enhanced visual feature sequence; and performing residual connection and layer normalization on the enhanced visual feature sequence and the visual feature sequence to obtain the fused visual feature sequence.

9. The emotion recognition method based on audiovisual collaboration according to claim 1, characterized in that, The temporal information aggregation of the fused visual feature sequence and the fused auditory feature sequence to obtain the audiovisual feature temporal aggregation vector includes: concatenating the features of the fused visual feature sequence and the fused auditory feature sequence to obtain the concatenated feature sequence; and inputting the concatenated feature sequence into a temporal aggregation model based on a bidirectional LSTM model to obtain the audiovisual feature temporal aggregation vector.

10. An emotion recognition system based on visual-auditory coordination, characterized in that, include: A multimodal data acquisition module is used to acquire raw video and raw audio streams; The visual feature encoding module is used to encode modality-specific features and uncertainties in the original video stream to obtain a visual feature sequence and a visual uncertainty sequence; the auditory feature encoding module is used to encode modality-specific features and uncertainties in the original audio stream to obtain an auditory feature sequence and an auditory uncertainty sequence; the multimodal cross-attention fusion module is used to perform uncertainty-guided cross-attention fusion of the visual feature sequence and the auditory feature sequence based on the auditory uncertainty sequence and the visual uncertainty sequence to obtain a fused visual feature sequence and a fused auditory feature sequence. The temporal information aggregation module is used to aggregate temporal information from the fused visual feature sequence and the fused auditory feature sequence to obtain a temporal aggregated vector of visual and auditory features; the emotion classification module is used to classify the temporal aggregated vector of visual and auditory features to obtain an emotion probability distribution.

Images