Emotion Recognition in Multimedia Videos Using Multimodal Fusion-Based Deep Neural Networks

Abstract

A landmark detection system and method using emotion recognition in multimedia videos using a multimodal fusion based deep neural network is provided. The system includes a circuit and a memory configured to store a multimodal fusion network including one or more feature extractors, a transformer encoder network, a fusion attention network, and an output network coupled to the fusion attention network. The system inputs a multimodal input to the one or more feature extractors. The multimodal input is related to an utterance depicted in the one or more videos. The system generates an input embedding as an output of the one or more feature extractors for the input, and further generates an emotion-related feature set based on the input embedding. The system further generates a fused feature representation of the emotion-related feature set, and predicts an emotion label of the utterance based on the fused feature representation.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS / INCORPORATION BY REFERENCE] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 263,961, filed November 12, 2021, which claims the benefit of priority to U.S. Patent Application No. 17 / 941,787, filed in the U.S. Patent and Trademark Office on September 9, 2022, the contents of which are incorporated herein by reference in their entireties.

[0002] Various embodiments of the present disclosure relate to neural networks and emotion recognition, and more particularly, to systems and methods for emotion recognition in multimedia videos using multimodal fusion based deep neural networks. [Background technology]

[0003] Advances in computer vision and artificial intelligence have led to the development of various kinds of neural networks (or models) that can be used in different applications, such as emotion recognition in conversations. Typically, emotion recognition is used to predict the emotional state of a speaker from a conversation(s) depicted in a multimedia video (e.g., movies, web series, and news). Emotion recognition is crucial in developing empathetic human-machine systems. In the case of conversations, traditional approaches for emotion recognition mostly rely on text transcripts of the conversation. Inaccuracies in the transcription can affect the accuracy of the predictions (e.g., emotion labels). Many state-of-the-art techniques for emotion recognition ignore the vast amount of information present in the visual and acoustic signals associated with the conversation. Summary of the Invention [Problem to be solved by the invention]

[0004] The limitations and disadvantages of the conventional approach will become apparent to one skilled in the art upon comparison of the described system with certain aspects of the present disclosure illustrated in the remaining portions of this application and with reference to the drawings. [Means for solving the problem]

[0005] Provided are systems and methods for emotion recognition in multimedia videos using multi-modal fusion based deep neural networks substantially as illustrated and / or described in connection with at least one of the drawings and more fully as set forth in the claims.

[0006] These and other features and advantages of the present disclosure will become apparent from a consideration of the following detailed description of the disclosure when taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a network environment for emotion recognition in multimedia videos using multimodal fusion-based deep neural networks, according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of an exemplary system for emotion recognition in multimedia videos using multimodal fusion based deep neural networks, in accordance with an embodiment of the present disclosure. [Diagram 3] FIG. 2 illustrates an example architecture of the multimodal fusion network of FIG. 1, in accordance with an embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates an exemplary visual feature extractor for the multi-modal fusion attention network of FIG. 3, in accordance with an embodiment of the present disclosure. [Diagram 5] FIG. 4 illustrates an example architecture of a fusion attention network of the set of fusion attention networks of FIG. 3, according to an embodiment of the present disclosure. [Figure 6] FIG. 2 illustrates an exemplary architecture of an audio-visual feature extractor of one or more feature extractors in accordance with an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates an example scenario for emotion recognition in multimedia videos using multimodal fusion-based deep neural networks, according to embodiments of the present disclosure. [Figure 8] 1 is a flowchart illustrating an exemplary method for emotion recognition in multimedia videos using multi-modal fusion based deep neural networks, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The disclosed system and method for emotion recognition in multimedia videos using multimodal fusion based deep neural networks may find implementation as described below. The system includes a circuit and a memory, the memory configured to store a multimodal fusion network including one or more feature extractors, a transformer encoder network coupled to the one or more feature extractors, a fusion attention network coupled to the transformer encoder network, and an output network coupled to the fusion attention network. The system may input a multimodal input to the one or more feature extractors. The multimodal input may relate to one or more utterances depicted in a video (such as a movie). The system may generate input embeddings as an output of the one or more feature extractors for the input. The input embeddings may include embeddings for each modality of the multimodal input. The system may further generate an emotion-related feature set based on applying the transformer encoder network to the input embeddings. The emotion-related feature set includes one or more features corresponding to each modality of the multimodal input. The system can further generate a fused-feature representation of the emotion-related feature set based on applying the fusion attention network to the emotion-related feature set. The system can predict an emotion label (e.g., angry, neutral, happy, sad, etc.) of the utterance based on applying the output network to the fused feature representation.

[0009] Emotions can be described as invisible mental states that can be related to the thoughts and feelings of a subject. In the absence of physiological indications, emotions can only be detected by human behaviors such as written speech, visual gestures, and acoustic signals. Conversational emotion recognition attempts to recognize the emotions of subjects in a conversation by relying on written, visual, and acoustic cues. Nowadays, conversational emotion recognition has become an essential task in the context of multimedia content analysis and moderation (such as video), helping to understand the nature of the interaction between users and content. Conversational emotion recognition has other important applications in AI interviews, personalized dialogue systems, opinion mining across chat histories, and many other tasks such as understanding user perception of content within social media platforms.

[0010] Current state-of-the-art emotion recognition methods frame the task of emotion recognition in a conversation as a purely text-based task. Specifically, current state-of-the-art emotion recognition methods in a conversation determine a subject's emotional state based on text data associated with the subject. The text data may correspond to a transcription of the subject's spoken audio. However, the vast amount of information present in the audio and visual modalities of multimedia content is not taken into account in determining the subject's emotional state.

[0011] The present disclosure provides a neural network architecture to detect a subject's emotional state using at least three different modalities related to speech: acoustic, text, and visual. Experimental data shows that a proper fusion of the three modalities can improve the quality and robustness of current state-of-the-art systems. The disclosed system can detect a subject's emotional state by taking as input each modality that contributes to the emotion prediction. The disclosed method can be more generalized compared to current state-of-the-art methods.

[0012] The present disclosure can also provide an acoustic-visual feature extractor that can be designed based on a triplet network to exploit the importance of triplet loss functions. The acoustic-visual feature extractor is trained on triplet loss functions, including an adaptive margin triplet loss function, a covariance loss function, and a variance loss function.

[0013] FIG. 1 illustrates a network environment for emotion recognition in multimedia videos using a multimodal fusion-based deep neural network, according to an embodiment of the present disclosure. FIG. 1 illustrates a diagram of a network environment 100. The network environment 100 includes a system 102. The system 102 includes a circuit 104 and a memory 106. The memory can include a multimodal fusion network 108. The multimodal fusion network 108 can include one or more feature extractors 110, a transformer encoder network 112, a fusion attention network 114, and an output network 116. FIG. 1 also illustrates a display device 118, a server 120, and a communication network 122. FIG. 1 also illustrates a multimodal input 124 and a predicted emotion label 126 displayed on the display device 118.

[0014] The circuitry 104 may include suitable logic, circuits, and interfaces that may be configured to execute program instructions associated with different operations to be performed by the system 102. The circuitry 104 may be implemented based on a number of processor technologies known in the art. Examples of processor technologies may include, but are not limited to, central processing units (CPUs), x86-based processors, reduced instruction set computing (RISC) processors, application specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, graphic processing units (GPUs), co-processors (such as inference accelerators or artificial intelligence (AI) accelerators), and / or combinations thereof.

[0015] The memory 106 may include suitable logic, circuitry, and / or interfaces that may be configured to store program instructions that the circuit 104 may execute. The memory 106 may also store the multimodal fusion network 108. In at least one embodiment, the memory 106 may also store input data for the multimodal fusion network 108, intermediate results obtained using the multimodal input embeddings, and emotion label(s) predicted by the multimodal fusion network 108. Example implementations of the memory 106 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and / or secure digital (SD) card.

[0016] The multimodal fusion network 108 can be a system of computational networks or artificial neurons arranged in layers. The multimodal fusion network 108 can be trained to predict emotion labels (e.g., emotion labels 126) of utterances depicted in one or more videos (i.e., emotion recognition in speech).

[0017] The multimodal fusion network 108 includes one or more feature extractors 110, a transformer encoder network 112 coupled to the one or more feature extractors 110, a fusion attention network 114 coupled to the transformer encoder network 112, and an output network 116 coupled to the fusion attention network 114. The operation of the multimodal fusion network 108 can be divided into two stages. In the first stage (i.e., utterance level), features can be extracted solely at the utterance level. Then, in the second stage (i.e., dialogue level), the network can learn to predict the emotion label of each utterance by using contextual information from the dialogue. An utterance can correspond to a short oral segment uttered by one of the parties in a multi-party conversation and can include transcripts, video clips, and audio segments. A dialogue can include multiple utterances in the order in which such utterances occurred in time.

[0018] Each of the one or more feature extractors 110 may be configured to perform one or more operations to generate an input embedding for each modality of the multimodal input (e.g., the multimodal input 124). Each encoder of the transformer encoder network 112 may be configured to perform one or more operations to generate an emotion-related feature set. The fusion attention network 114 may be configured to generate a fused feature representation of the emotion-related feature set. Specifically, the fusion attention network 114 may be configured to generate the fused feature representation based on applying one or more multi-head attention operations to the emotion-related feature set. The output network 116 may be configured to predict an emotion label 126 of an utterance associated with the multimodal input 124. The output network 116 may predict the emotion label 126 based on the fused feature representation. The output network 116 may include a fully connected layer that may be configured to predict the emotion label 126. Details of the output network 116 are shown, for example, in FIG. 3.

[0019] Each of the one or more feature extractors 110, each encoder of the transformer encoder network 112, the fusion attention network 114, and the output network 116 can be a system of neural networks or artificial neurons that can be arranged in multiple layers. The multiple layers of the neural network can include an input layer, one or more hidden layers, and an output layer. Each layer of the multiple layers can include one or more nodes (i.e., artificial neurons). The output of every node in the input layer can be coupled to at least one node in the hidden layer(s). Similarly, the input of each hidden layer can be coupled to the output of at least one node in another layer of the neural network. The output of each hidden layer can be coupled to the input of at least one node in another layer of the neural network. The node(s) in the final layer can receive inputs from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer can be determined from hyperparameters of the neural network. Such hyperparameters can be set before or after training the neural network on a training dataset. In the multimodal fusion network 108, a training dataset can include a set of multimodal inputs and corresponding emotion labels as ground truth values. Each multimodal input can include at least one of audio of an utterance, one or more frames in a scene in which one or more characters produce the utterance, and a transcription of the audio.

[0020] Each node of a neural network can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the network. The set of parameters can include, for example, weight parameters and regularization parameters. Each node can calculate an output using a mathematical function based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the neural network. All or some of the nodes of a neural network can correspond to the same or different mathematical functions.

[0021] In training a neural network, one or more parameters of each node of the neural network may be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on the neural network's loss function. The above process may be repeated for the same or different inputs until a minimum of the loss function is achieved and the training error is minimized. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristic methods.

[0022] Each of the one or more feature extractors 110, each encoder of the transformer encoder network 112, the fusion attention network 114, and the output network 116 may include electronic data that may be implemented, for example, as a software component of an application executable on the system 102. Each of the one or more feature extractors 110, each encoder of the transformer encoder network 112, the fusion attention network 114, and the output network 116 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device, such as the circuit 104. Each of the encoders of the transformer encoder network 112, the fusion attention network 114, and the output network 116 may include code and routines configured to enable a computing device, such as the circuit 104, to perform one or more operations. For example, each of the one or more feature extractors 110 may perform one or more operations to generate an input embedding for each modality of the received multimodal input 124. Each encoder of the transformer encoder network 112 may perform one or more operations to generate an emotion-related feature set. Additionally or alternatively, each of the one or more feature extractors 110, each encoder of the transformer encoder network 112, the fusion attention network 114, and the output network 116 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling one or more operations), a tensor processing unit (TPU), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, each of the one or more feature extractors 110, each encoder of the transformer encoder network 112, the fusion attention network 114, and the output network 116 may be implemented using a combination of hardware and software.

[0023] In one embodiment, each encoder of the transformer encoder network 112 can be configured to receive input embeddings of each modality in parallel (i.e., simultaneously) as input and simultaneously provide emotion-related feature sets as output. By way of example and not limitation, each encoder can include a multi-head attention layer and a feedforward neural network.

[0024] In one embodiment, a fusion attention network 114 can be used to incorporate visual and acoustic modalities into the textual modality associated with the speech. The fusion attention network 114 can include one or more multi-head attention layers and a first fully connected layer. Details of the fusion attention network are shown, for example, in FIG. 5.

[0025] The display device 118 may include suitable logic, circuitry, and interfaces that may be configured to display emotion labels 126 of utterances associated with the multimodal input 124. In some embodiments, the display device 118 may be configured to display the multimodal input 124 and emotion labels 126 corresponding to the utterance-level portions of the multimodal input 124. The display device 118 may be realized through a number of known technologies, such as, but not limited to, at least one of a liquid crystal display (LCD) display, a light-emitting diode (LED) display, a plasma display, or an organic LED (OLED) display technology, or other display device. According to some embodiments, the display device 118 may refer to a display screen of a head-mounted device (HMD), a smart glasses device, a see-through display, a projection display, an electrochromic display, or a transparent display.

[0026] In another embodiment, the display device 118 may include suitable logic, circuitry, interfaces, and / or code to enable the multimodal fusion network 108 to be implemented as part of a software program or service (such as an application programming interface (API) based service) executable on the display device 118. The multimodal fusion network 108 may be implemented on the display device 118 after training of the multimodal fusion network 108 on the system 102 is completed. Examples of the display device 118 may include, but are not limited to, a computing device, a mainframe machine, a server, a computer workstation, a smart phone, a mobile phone, a gaming device, a wearable display, a consumer electronics (CE) device, and / or any other device having image processing capabilities.

[0027] The server 120 may include suitable logic, circuits and interfaces, and / or code that may be configured to store one or more videos for emotion recognition and other operations, such as media streaming operations. The server 120 may also be configured to store predictive emotion labels 126 or speech level portions of the videos. The server 120 may be implemented as a cloud server and may perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, and the like. Other implementations of the server 120 may include, but are not limited to, a database server, a file server, a web server, an application server, a mainframe server, or a cloud computing server.

[0028] In at least one embodiment, server 120 can be implemented as multiple distributed cloud-based resources using a number of techniques known to those of skill in the art. Those skilled in the art will appreciate that the scope of the present disclosure may not be limited to the implementation of server 120 and system 102 as two independent entities. In some embodiments, the functionality of server 120 can be incorporated in whole or at least in part into system 102 without departing from the scope of the present disclosure.

[0029] The communication network 122 may include a communication medium that allows the system 102, the display device 118, and the server 120 to communicate with each other. The communication network 122 may include either a wired connection or a wireless connection. Examples of the communication network 122 may include, but are not limited to, the Internet, a cloud network, a cellular or wireless mobile network (such as long-term evolution or 5G new wireless), a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 122 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE802.11, Light Fidelity (Li-Fi), 802.16, IEEE802.11s, IEEE802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0030] In operation, the circuit 104 can be configured to input a multimodal input 124 to one or more feature extractors 110. The multimodal input can be related to speech depicted in one or more videos. For example, the multimodal input can include a first modality related to the acoustics of the speech, a second modality related to a transcription of the speech, and a third modality related to visual aspects of the speech. In an embodiment, the multimodal input can further include a fourth modality that can be related to one or more biological parameters of a subject (i.e., the speaker) associated with the corresponding speech.

[0031] The circuit 104 may be configured to generate an input embedding as an output of one or more feature extractors 110 for the input. The input embedding may include an embedding for each modality of the multimodal input 124. In an embodiment, the input embedding may correspond to features of the corresponding modality. Specifically, the input embedding may be referred to as a low-dimensional learned continuous vector representation of the discrete variables. Based on the generation of the input embedding, the circuit 104 may be further configured to generate an emotion-related feature set based on applying a transformer encoder network 112 to the input embedding. In an embodiment, the transformer encoder network 112 may be applied to the input embedding to learn the context of the corresponding utterance for each modality. The emotion-related feature set may include one or more features corresponding to each modality of the multimodal input 124. More details regarding the emotion-related feature set are shown, for example, in FIG. 3.

[0032] To map each modality into a corresponding text vector space, the circuitry can be configured to apply a fusion attention network 114 to the emotion-related feature set. In another embodiment, the fusion attention network 114 can be used to incorporate visual and acoustic information related to the dialogue. In particular, the circuitry 104 can be further configured to generate a fusion feature representation of the emotion-related feature set based on applying the fusion attention network 114 to the emotion-related feature set. More details regarding the fusion feature representation are shown, for example, in FIG. 3.

[0033] After generating the fused feature representation, the circuitry 104 may be configured to predict an emotion label 126 of an utterance associated with the multimodal input 124. The emotion label 126 may be predicted based on applying the output network 116 to the fused feature representation. In one embodiment, the circuitry 104 may be further configured to control the display device 118 to render the predicted emotion label 126 on the display device 118. More details regarding the emotion label 126 and the prediction of the emotion label 126 are shown, for example, in FIG. 3.

[0034] FIG 2 is an example block diagram of a system for emotion recognition in multimedia videos using a multimodal fusion based deep neural network, according to an embodiment of the present disclosure. The description of FIG 2 is provided with reference to the elements of FIG 1. FIG 2 illustrates a block diagram 200 of the system 102 of FIG 1. The system includes a circuit 104, a memory 106, a multimodal fusion network 108, an input / output (I / O) device 202, a network interface 204, an inference accelerator 206, a translator model 208, a face detection model 210, a scene detection model 212, and a single boundary detection model 214.

[0035] The I / O devices 202 may include suitable logic, circuitry, and / or interfaces that may be configured to receive one or more user inputs and / or render information (such as the predicted emotion labels 126) generated by the system 102. The I / O devices 202 may include a variety of input / output devices that may be configured to communicate with different operational components of the system 102. Examples of the I / O devices 202 may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a microphone, and a display device (such as display device 118).

[0036] The network interface 204 may include suitable logic, circuitry, interfaces, and / or code that may be configured to establish communications between the system 102, the display device 118, and the server 120 over the communications network 122. The network interface 204 may be configured to implement known technologies that support wired or wireless communications. The network interface 204 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0037] The network interface 204 can be configured to communicate via offline and online wireless communications with networks such as the Internet, an intranet, and / or wireless networks such as a cellular telephone network, a wireless local area network (WLAN), a personal area network, and / or a metropolitan area network (MAN). The wireless communications can use any of a number of communication standards, protocols, and technologies such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), LTE, 5G New Wireless, Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (WiFi) (such as IEEE 802.11, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or any other IEEE 802.11 protocol), Voice over Internet Protocol (VoIP), Wi-MAX, Internet of Things (IoT) technologies, Machine Type Communication (MTC) technologies, protocols for email, instant messaging, and / or short message service (SMS).

[0038] The inference accelerator 206 may include suitable logic, circuitry, interfaces, and / or code that may be configured to operate as a co-processor of the circuitry 104 to accelerate computations associated with the operation of the multimodal fusion network 108. For example, the inference accelerator 206 may accelerate computations on the system 102 such that an emotion label 126 is predicted in a shorter time than would normally occur without the use of the inference accelerator 206. The inference accelerator 206 may implement various acceleration techniques, such as parallelizing the operation of some or all of one or more of the feature extractor 110, the transformer encoder network 112, the fusion attention network 114, and the output network 116. The inference accelerator 206 may be implemented as software, hardware, or a combination thereof. Examples of implementations of the inference accelerator 206 include, but are not limited to, a GPU, a tensor processing unit (TPU), a neuromorphic chip, a vision processing unit (VPU), a field programmable gate array (FPGA), a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, a microcontroller, and / or combinations thereof.

[0039] The translator model 208 may include suitable logic, circuitry, interfaces, and / or code that may be configured to translate speech in a second language into a first language (or vice versa). In an embodiment, the translator model 208 may be configured to translate a transcript of speech in a first language into a second language. Examples of the translator model 208 may include, but are not limited to, an artificial neural network (ANN), a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a gated recurrent unit (GRU)-based RNN, a CNN-recurrent neural network (CNN-RNN), a long short-term memory (LSTM) network-based RNN, LSTM+ANN, and / or a combination of such networks.

[0040] The face detection model 210 may include suitable logic, circuitry, interfaces, and / or code that may be configured to detect one or more faces in an image (or frame). The face detection model 210 may detect one or more faces in an image using one or more face detection techniques. Detailed implementations of one or more face detection techniques are believed to be familiar to those skilled in the art, and thus, a detailed description of the one or more face detection techniques described above is omitted from this disclosure for the sake of brevity. Examples of the face detection model 210 may include, but are not limited to, a Convolutional Neural Network (CNN), R-CNN, Faster R-CNN, (You Only Look Once) YOLO network, and / or combinations of such networks.

[0041] The scene detection model 212 may include suitable logic, circuitry, interfaces, and / or code that may be configured to extract multiple scenes from one or more videos. In some embodiments, the multiple scenes may be extracted based on background pixel information (e.g., changes in background pixel values). Examples of the scene detection model 212 may include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), an artificial neural network (ANN), and / or a combination of such networks.

[0042] The single boundary detection model 214 may include suitable logic, circuitry, interfaces, and / or code that may be configured to detect multiple speeches in a scene. In one embodiment, the single boundary detection model 214 may include a VGG-16 convolutional neural net (CNN) followed by multiple convolutional layers. The VGG-16 may be used for feature extraction, and the convolutional layers may be used for object detection. The single boundary detection model 214 may further extract multiple scenes of the video based on the detection of the object. In one embodiment, the single boundary detection model 214 may extract multiple scenes of the video based on background pixel information (e.g., changes in background pixel values).

[0043] Figure 3 illustrates an example architecture of the multimodal fusion network of Figure 1, according to an embodiment of the present disclosure. The description of Figure 3 is provided with reference to elements of Figures 1 and 2. Figure 3 illustrates a diagram 300 of a multimodal fusion network 302, which may be an example implementation of the multimodal fusion network 108 of Figure 1.

[0044] The multimodal fusion network 302 may include one or more feature extractors 304, a transformer encoder network 306, a set of fusion attention networks 308, and an output network 310. The transformer encoder network 306 may be coupled to the one or more feature extractors 304, a first fusion attention network 308A of the set of fusion attention networks 308 may be coupled to the transformer encoder network 306, and the output network 310 may be coupled to an Nth fusion attention network 308N of the set of fusion attention networks 308. According to an embodiment, there may be a single fusion attention network (e.g., the first fusion attention network 308A) that may be coupled to the transformer encoder network 306 and the output network 310. Figure 3 further illustrates a block-style representation of multiple utterances 312 corresponding to a video or a conversation depicted in one or more videos.

[0045] At any point in time, the circuit 104 may receive one or more videos that may depict a conversation between multiple human speakers or characters (e.g., a dyadic conversation). For each utterance in such a conversation, an emotion label needs to be predicted. The operation of predicting emotion labels is described herein.

[0046] The circuit 104 may be configured to apply a scene detection model 212 to one or more frames of the video after it is received. As described in FIG. 2, the scene detection model 212 may be trained to extract multiple scenes (or multiple dialogues) from each video. The circuit 104 may process frames of the video corresponding to multiple scenes for automatic detection of speech. By way of example and not limitation, a single boundary detection model 214 may be applied to the frames to detect multiple objects in each such frame. Based on this detection, the circuit 104 may further detect a subset of frames corresponding to multiple utterances 312.

[0047] The circuitry 104 can extract audio portions from one or more videos. Each such audio portion can include speech sounds corresponding to speech (as part of a conversation or dialogue). Similarly, the circuitry 104 can generate transcriptions of such audio portions using suitable Speech-To-Text (STT) techniques.

[0048] The plurality of utterances 312 may include a first utterance 312A, a second utterance 312B, a third utterance 312C..., and a Kth utterance 312K. An utterance (such as the first utterance 312A) may be defined as a part of a dialogue or conversation that may be represented through a combination of speech sounds, an image (or images), and a transcription of the speech sounds. Each of the plurality of utterances 312 may have a corresponding first modality related to the acoustics of the corresponding utterance, a second modality of the plurality of modalities related to a transcription of the corresponding utterance, and a third modality related to visual aspects of the corresponding utterance (e.g., facial expressions and lip movements, etc.). For example, a first utterance 312A may include a first modality 314A related to the acoustics of the first utterance 312A, a second modality 314B related to a transcription of the first utterance 312A, and a third modality 314C related to visual aspects of the first utterance 312A. The first modality 314A, the second modality 314B, and the third modality 314C may collectively form a multimodal input (such as multimodal input 124).

[0049] In one embodiment, a scene (or dialogue) can include "k" utterances "U" and their respective emotion labels "Y" that can be arranged together in time. Each utterance can be accompanied by a corresponding first modality (i.e., speech segment), a second modality (i.e., transcription), and a third modality (i.e., video clip). As an example, a scene of "k" utterances can be mathematically represented using the following equation (1): TIFF2024546432000002.tif13150(1) Where: x i represents the i-th utterance, TIFF2024546432000003.tif7150 represents the acoustic data associated with the i-th utterance, TIFF2024546432000004.tif13150 represents the transcription associated with the i-th utterance, TIFF2024546432000005.tif7150 represents the video associated with the i-th utterance, y i represents the emotion label of the i-th utterance.

[0050] The circuit 104 may be configured to input a multi-modal input to one or more feature extractors 304. For example, the multi-modal input may relate to a first utterance 312A depicted in one or more received videos.

[0051] According to one embodiment, the multimodal input may include multilingual speech and a transcription of the multilingual speech in a first language that may be adapted to one or more feature extractors 304. For example, the multilingual speech may correspond to the utterance "Hello Fred, Genkidesu ka?" In such a case, the transcription of the multilingual speech may be "Hello Fred, how are you?"

[0052] According to another embodiment, the multimodal input may include speech in a second language, which may be different from a first language that is compatible with the one or more feature extractors 304. In such a case, the multimodal input may include a transcription of the speech in the first language (compatible with the one or more feature extractors 304). In such an embodiment, the circuit 104 may be configured to apply the translator model 208 to the speech (in the second language) to translate the speech from the second language to the first language. The translation may be performed to overcome language compatibility issues between the speech (in the second language) and the one or more feature extractors 304.

[0053] The one or more feature extractors 304 may include an acoustic feature extractor 304B, a text feature extractor 304C, a visual feature extractor 304D, and an audio-visual feature extractor 304C. In an embodiment, each of the one or more feature extractors 304 may include at least one neural network that may be configured to extract features related to a corresponding modality. For example, the acoustic feature extractor 304B and the audio-visual feature extractor 304C may be configured to extract features related to the acoustics of the first utterance 312A. Similarly, the text feature extractor 304C may be configured to extract features related to a transcription of the first utterance 312A, and the visual feature extractor 304D and the audio-visual feature extractor 304C may be configured to extract features related to visual aspects of the first utterance 312A. Collectively, such features may be referred to as input embeddings.

[0054] In an embodiment, the circuit 104 can be configured to generate an input embedding based on applying one or more feature extractors 304 to the multimodal input. In particular, the circuit 104 can be configured to generate a first embedding (FIA) based on applying an audiovisual feature extractor 304C to the acoustic information of the speech included in the multimodal input. The audiovisual feature extractor 304C can be based on a triplet network that allows the audiovisual feature extractor 304C to exploit the importance of three loss functions. The audiovisual feature extractor 304C can include an encoder network and a projector module, and can be trained for loss functions such as adaptive margin triplet loss, covariance loss, and variance loss. Details of the audiovisual feature extractor 304C are shown, for example, in FIG. 6.

[0055] In an embodiment, the circuit 104 may be configured to perform one or more operations on the audio information before the audio information is provided as input to the audio-visual feature extractor 304C. The circuit 104 may be configured to convert the audio portion included in the audio information into a two-dimensional (2D) Mel Spectrogram in RGB format. Such a two-dimensional Mel Spectrogram in RGB format may be provided as input to the audio-visual feature extractor 304C. To convert the audio portion into a two-dimensional (2D) Mel Spectrogram, the circuit 102 may be configured to process the audio signals of the audio portion through one or more enhancement techniques, such as time warping and Additive White Gaussian Noise (AWGN), to generate augmented audio signals. Such generated augmented signals may be further converted into Mel Spectrograms. In one embodiment, the mel spectrogram can be calculated using a short-time Fourier transform (STFT) with a frame length of 400 samples (25 ms) and a hop length of 160 samples (10 ms) and a 128 mel filter bank. The circuit performs a first embedding (F IA ) may be further configured to generate

[0056] In another embodiment, the circuit 104 is configured to generate a first embedding (F IA) based on the openSMILE (open-source Speech and Music Interpretation by Large-space Extraction) model. In one embodiment, the acoustic feature extractor 304B may further include a multi-layer perceptron (MLP) network that may be trained on the utterance labels. The circuit 104 may generate an embedding (F) of the input embedding based on applying a multi-layer perceptron (MLP) network to the output generated by the acoustic feature extractor 304B using openSMILE. IA For example, the first embedding may be mathematically represented using the following equation (2): TIFF2024546432000006.tif13150(2) Where: TIFF2024546432000007.tif13150 represents the first embedding, TIFF2024546432000008.tif7150 represents the operation of audio-video feature extractor 304C or audio feature extractor 304B; k represents the total number of multiple utterances, TIFF2024546432000009.tif7150 represents the acoustic (or audio component) of the i-th utterance, D A represents the size of the embedding of the speech utterance.

[0057] In an embodiment, the circuit 104 can be configured to generate a second embedding of the input embedding based on applying the text feature extractor 304C to a transcription of the acoustic information associated with the first utterance 312A. The second embedding can be generated further based on applying the text feature extractor 304C to a transcription of a different utterance that precedes or follows in time the first utterance 302A. The transcription of the different utterance that precedes or follows in time the first utterance 302A can include a separator token ( <s>For example, if the fourth utterance in a scene is "The whole thing! Can we go?", the fifth utterance in the scene is "What about the scene with the Kangaroo", and the sixth utterance in the scene is "I was surprised to see a Kangaroo in a world war epic", the transcription of the fifth utterance will be "The whole thing! <s>What about the scene with the Kangaroo <s>I was surprised to see a Kangaroo in a world war epic.

[0058] In one embodiment, the text feature extractor 304C can be implemented based on the Robustly optimized Bidirectional Encoder Representations from Transformers (BERT) model. For example, the second embedding can be mathematically represented using the following equation (3): TIFF2024546432000010.tif13150(3) Where: TIFF2024546432000011.tif13150 represents the second embedding of the input embedding, TIFF2024546432000012.tif7150 represents the operation of the text feature extractor 304C (RoBERTa model or modified RoBERTa model), k represents the total number of multiple utterances, TIFF2024546432000013.tif13150 represents the transcription of the i-th utterance, D T represents the embedding size of the text utterance.

[0059] In an embodiment, the circuit 104 can be configured to generate a third embedding of the input embedding based on applying one of the audiovisual feature extractor 304C or the visual feature extractor 304D to face information of one or more speaking characters in frames of one or more videos and scene information related to the frames. The frames can correspond to the duration of the first utterance 312A in the one or more videos. Each of the audiovisual feature extractor 304C and the visual feature extractor 304D can be a dual network that can be configured to detect, for example, one or more faces of one or more speaking characters and respective areas of the one or more faces. In an embodiment, the audiovisual feature extractor 304C or the visual feature extractor 304D can include a first network that extracts features from the detected one or more features of the one or more speaking characters and a second network that extracts features from an entire scene including the one or more speaking characters. The visual feature network can normalize the detected face or faces based on their corresponding areas to generate a third embedding of the input embedding. Details of the audiovisual feature extractor 304C and the visual feature extractor 304D are shown, for example, in FIG. 4. The generated third embedding can be mathematically represented using the following equation (4): TIFF2024546432000014.tif13150(4) Where: TIFF2024546432000015.tif13150 represents the third embedding of the input embeddings, TIFF2024546432000016.tif7150 represents the operation of the audiovisual feature extractor 304C or the visual feature extractor 304D; k represents the total number of multiple utterances, TIFF2024546432000017.tif7150 represents the video associated with the i-th utterance, D V represents the embedding size of the visual utterance.

[0060] The generated input embeddings (including embeddings for each modality of the multimodal input) can be provided as inputs to a transformer encoder network 306. The input embeddings can be passed through the transformer encoder network 306 to learn inter-utterance context for each modality of the multimodal input. The transformer encoder network 306 can include a first stack of transformer encoders 316 for a first modality of the multimodal input 314A, a second stack of transformer encoders 318 for a second modality of the multimodal input 314B, and a third stack of transformer encoders 320 for a third modality of the multimodal input 314C. In an embodiment, the first stack of transformer encoders 316, the second stack of transformer encoders 318, and the third stack of transformer encoders 320 can include the same number of transformer encoders. In another embodiment, the first stack of transformer encoders 316, the second stack of transformer encoders 318, and the third stack of transformer encoders 320 may include a different number of transformer encoders.

[0061] The first stack of transformer encoders 316 may include a first transformer encoder 316A... through an Nth transformer encoder 316N. An output of the first transformer encoder 316A may be provided as an input to the Nth transformer encoder 316N. Similarly, the second stack of transformer encoders 318 may include a first transformer encoder 318A... through an Nth transformer encoder 318N. An output of the first transformer encoder 318A may be provided as an input to the Nth transformer encoder 318N. Similarly, the third stack of transformer encoders 320 may include a first transformer encoder 320A... through an Nth transformer encoder 320N. An output of the first transformer encoder 320A may be provided as an input to the Nth transformer encoder 320N.

[0062] The first embedding of the input embedding (F IA ) may be provided as an input to a first transformer encoder 316A of a first stack of transformer encoders 316. A second embedding of the input embedding (F IT ) may be provided as an input to a first transformer encoder 318A of a second stack of transformer encoders 318. Similarly, a third embedding of the input embedding (F IV ) may be provided as an input to a first transformer encoder 318A of a third stack 320 of transformer encoders.

[0063] Each transformer encoder of the transformer encoder network 306 can be trained to generate an emotion-related feature set. The emotion-related feature set can include one or more features corresponding to each modality of the multi-modal input. For example, the one or more features corresponding to the first modality 314A can be mathematically represented using the following equation (5): TIFF2024546432000018.tif13150(5) Where: TIFF2024546432000019.tif13150 represents the first embedding of the input embeddings, T δ represents the operation of the transformer encoder, N 1 represents the first transformer encoder 316A of the first stack of transformer encoders 316, N A represents the Nth transformer encoder 316N of the first stack of transformer encoders 316, For i∈[1,k].

[0064] The one or more features corresponding to the second modality 314B may be mathematically represented using the following equation (6): TIFF2024546432000020.tif13150(6) Where: TIFF2024546432000021.tif13150 represents the second embedding of the input embedding, T δ represents the operation of the transformer encoder, N 1 represents the first transformer encoder 318A of the second stack of transformer encoders 318, N T represents the Nth transformer encoder 318N of the second stack of transformer encoders 318, For i∈[1,k]. Similarly, one or more features corresponding to the third modality 314C may be mathematically represented using the following equation (7): TIFF2024546432000022.tif13150(7) Where: TIFF2024546432000023.tif13150 represents the third embedding of the input embeddings, T δ represents the operation of the transformer encoder, N 1 represents the first transformer encoder 320A of the third stack of transformer encoders 320, N V represents the Nth transformer encoder 318N of the third stack of transformer encoders 320, For i∈[1,k].

[0065] In an embodiment, the multimodal fusion network 302 may further include a skip connection 322 between each pair of adjacent transformer encoders in the transformer encoder network 306. Specifically, the skip connection 322 may be present between each pair of adjacent transformer encoders in the first stack of transformer encoders 316, the second stack of transformer encoders 318, and the third stack of transformer encoders 320. The skip connection 322 may be employed in the multimodal fusion network 302 to prevent the multimodal fusion network 302 from ignoring lower level features associated with each modality of the multimodal input.

[0066] The generated emotion-related feature set can be provided as input to a fusion attention network 308 of a set of fusion attention networks 308. The set of fusion attention networks 308 can be coupled to the transformer encoder network 306 and can include at least one fusion attention network. By way of example, the set of fusion attention networks 308 can include, but is not limited to, a first fusion attention network 308A... through an Nth fusion attention network 308N. Each fusion attention network of the set of fusion attention networks 308 can include one or more multi-head attention layers and a first fully connected layer. In some embodiments, the input of the first fully connected layer can be coupled to the output of one or more multi-head attention layers of the corresponding fusion attention network. Each of the set of fusion attention networks 308 can be configured to output a fused feature representation of the emotion-related feature set. Details regarding each of the set of fusion attention networks 308 and the emotion-related feature sets are shown, for example, in FIG. 5. By way of example and not limitation, the fused feature representation of the emotion-related feature set may be mathematically represented using the following equations (8) and (9): TIFF2024546432000024.tif13150(8) TIFF2024546432000025.tif13150(9) Where: TIFF2024546432000026.tif13150 represents the output of the first fusion attention network 308A, MHA represents the operation of one or more multi-head attention layers, TIFF2024546432000027.tif13150 represents the output of the Nth fusion attention network 308N (or the output of the set of fusion attention networks 308), TIFF2024546432000028.tif13150 represents one or more features corresponding to a first modality 314A; TIFF2024546432000029.tif13150 represents one or more features corresponding to a second modality 314B; TIFF2024546432000030.tif13150 represents one or more features corresponding to a third modality 314C; m represents the total number of one or more multi-head attention layers in the fusion attention network set 308; For i∈[1, k].

[0067] In one embodiment, the generated fused feature representation can be provided as an input to an output network 310. The output network 310 can include a second fully connected layer that can be coupled to an output of the fusion attention network set 308. The second fully connected layer of the output network 310 can be configured to predict an emotion label of a first utterance 312A of the plurality of utterances 312. In one embodiment, the second fully connected layer can include a SoftMax function or a cross-entropy function implemented on an output of the second fully connected layer. The predicted emotion label can be one of, but is not limited to, a happy emotion label, a sad emotion label, an angry emotion label, a calm emotion label, a fearful emotion label, a neutral emotion label, an excited emotion label, a confused emotion label, a stressed emotion label, an annoyed emotion label, a surprised emotion label, an excited emotion label, or a frightened emotion label.

[0068] In one embodiment, the output network 310 can be configured to predict a sentiment label of the multimodal input. The predicted sentiment label can indicate whether an utterance (for which the multimodal input is provided as input to the multimodal fusion network 302) corresponds to a positive sentiment, a negative sentiment, or a neutral sentiment.

[0069] In one embodiment, the output of the output network 310 may be mathematically represented using the following equations (10) and (11): TIFF2024546432000031.tif13150(10) TIFF2024546432000032.tif13150(11) Where: TIFF2024546432000033.tif13150 represents predicted emotion labels for each of a number of utterances 312; TIFF2024546432000034.tif13150 represents a predicted emotion label for a first utterance 312A of the plurality of utterances 312; TIFF2024546432000035.tif13150 represents a predicted emotion label for the pth utterance among the plurality of utterances 312; FC represents the operation of the second fully connected layer of the output network 310; TIFF2024546432000036.tif13150 represents the output of the Nth fusion attention network 308N (or the output of the set of fusion attention networks 308), For i∈[1,k].

[0070] In one embodiment, the operation of the multimodal fusion network 302 can be divided into two levels: an utterance level and a dialogue level. The one or more feature extractors 304 can be considered as part of the utterance level, since embeddings associated with each modality of the multimodal input can be generated independently. At the dialogue level, the multimodal fusion network 302 can learn to predict the sentiment of each utterance by using contextual information from the entire dialogue.

[0071] In an embodiment, the multiple modalities may also include a fourth modality related to one or more biological parameters of the subject associated with the corresponding utterance. For example, the one or more biological parameters may be captured during the recording of the utterance and converted into time series data (or multi-dimensional spatial data) for further processing by the multi-modal fusion network 302. The circuit 104 may be configured to predict an emotion label of the subject based on all four modalities, i.e., the first modality, the second modality, the third modality, and the fourth modality. Below is a pseudocode of the operations performed by the system 102: TIFF2024546432000037.tif149150, where MHA supports multi-head attention network operation (i.e., fusion attention network operation), Concat corresponds to the concatenation operation, FC corresponds to the operation of the second fully connected layer, D T corresponds to the size of the text.

[0072] FIG. 4 illustrates an example visual feature extractor of the multimodal fusion attention network of FIG. 3 according to an embodiment of the present disclosure. FIG. 4 is described in conjunction with elements of FIG. 1, FIG. 2, and FIG. 3. FIG. 4 illustrates a diagram 400 that may include a system 402. The system 402 may be an example implementation of the system 102 of FIG. 1. The system 402 includes a multimodal fusion network 302. The multimodal fusion network 302 may include one or more feature extractors 304. FIG. 4 further illustrates a frame 404 of the plurality of frames that corresponds to a duration of a first utterance 312A of the plurality of utterances 312.

[0073] The system 402 can input one or more frames of a video to a visual feature extractor 304D. The visual feature extractor 304D can include a face detection model 210. In one embodiment, the face detection model 210 can correspond to a Multi-task Cascaded Convolutional Network (MTCNN). The face detection model 210 can be applied to the frame 414 to detect one or more faces in each of the received frames. As an example, in the frame 404, the one or more detected faces can include a first face 406 and a second face 408.

[0074] The system 402 can be configured to generate one or more bounding boxes that can contain the detected face or faces. The generated one or more bounding boxes can include a first bounding box 410 and a second bounding box 412. The first bounding box 410 can include a first face 406, and the second bounding box 412 can include a second face 408. In an embodiment, the one or more bounding boxes can be generated based on applying the face detection model 210 to the frame 404. The system 402 can then normalize an area associated with each of the one or more bounding boxes based on the application of one of the audio-visual feature extractor 304C or the visual feature extractor 304D. The system 402 can generate a third embedding of the input embedding as an output of the audio-visual feature extractor 304C or the visual feature extractor 304D based on the normalization.

[0075] The system 402 may be configured to determine a weighted sum based on features associated with each of the one or more detected faces and the corresponding normalized area. The weighted sum may be mathematically represented using the following equation (12): TIFF2024546432000038.tif7150(12) Where: F IV represents the third embedding of the input embedding, F 1 represents features associated with the first detected face 406, F 2 represents features associated with the detected second face 408, W 1 represents the normalized area of ​​the first bounding box 410, W 2 represents the normalized area of ​​the second bounding box 412.

[0076] In one embodiment, the visual feature extractor 304D may include a visual transformer (ViT) that may be used on a first total number of consecutive frames (e.g., 15 for the audio-visual feature extractor 304C and 30 for the visual feature extractor 304D) that spans the duration of the first utterance 312A. The system 102 may extract features from each frame in the first total number based on the operations described above. The features extracted from each of the first total number of consecutive frames may be max pooled to generate a third embedding (expressed using equations (7) and (12)).

[0077] In another embodiment, the audiovisual feature extractor 304C or the visual feature extractor 304D can be a dual network. The dual network can include a first network that detects one or more faces in the frame 404 and a second network that focuses on the entire frame 404. In particular, the second network can focus on one or more objects and other visual cues (i.e., scene information) visible in the frame 404. The circuit 104 can be further configured to generate a third embedding of the input embedding as an output of the audiovisual feature extractor 304C or the visual feature extractor 304D based on applying the first network and the second network to the frame(s).

[0078] Figure 5 illustrates an example architecture of the fusion attention network of Figure 3, according to an embodiment of the present disclosure. The description of Figure 5 is provided with reference to elements of Figures 1, 2, 3, and 4. Figure 5 illustrates a diagram 500 of a first attention network 502, which may be an example embodiment of the first fusion attention network 308A of the set of fusion attention networks 308.

[0079] The first attention network 502 may include one or more multi-head attention layers, which may further include a first multi-head attention layer 504A and a second multi-head attention layer 504B. The first attention network 502 may further include a fully connected layer 506 (also referred to as a first fully connected layer). The architecture of each fusion attention network in the fusion attention network set 308 may be the same as that shown in FIG. 500. The fusion attention network set 308 may be coupled to a transformer encoder network 306 and an output network 310. Specifically, the output of the transformer encoder network 306 (i.e., the emotion-related feature set) may be provided as an input to the fusion attention network set 308 (specifically, the first fusion attention network 308A), and the output of the fusion attention network set 308 may be provided as an input to the output network 310.

[0080] The circuit 104 may be configured to provide the emotion-related feature set as an input to a first attention network 502 of the set of fusion attention networks 308. As described above, the emotion-related feature set may include one or more features associated with a first modality (represented by equation (5)), one or more features associated with a second modality (represented by equation (6)), and one or more features associated with a third modality (represented by equation (7)). Specifically, the circuit 104 may be configured to provide the emotion-related feature set to one or more multi-head attention layers of the first attention network 502.

[0081] Each of the one or more multi-head attention layers can accept a query, a key, and a value as input and can be configured to capture various ranges of dependencies (e.g., short range and long range) in the sequence. In one embodiment, one or more features (F A ) as a "key (k)" and one or more features (F T ) can be provided to the first multi-head attention layer 504A as a “query (q) and value (v).” Similarly, one or more features (F V ) as a "key (k)" and one or more features related to the third modality (F T ) can be provided to the second multi-head attention layer 504B as “query (q) and value (v).”

[0082] The circuit 104 may be configured to apply one or more multi-head attention layers to the emotion-related feature set to determine feature-to-feature mappings within the emotion-related feature set. As described, each of the one or more multi-head attention layers may capture dependencies between feature(s) associated with a first modality, feature(s) associated with a second modality, and feature(s) associated with a third modality. The mapping may be used to map each respective modality of the multiple modalities to a text vector space. The circuit 104 may be configured to concatenate the emotion-related feature set to a latent representation of the emotion-related feature set based on the feature-to-feature mappings. After concatenation, the concatenated output (i.e., the latent representation of the emotion-related feature set) may be provided as an input to the fully connected layer 506. The circuit 104 may be configured to generate a fused feature representation of the emotion-related feature set based on the application of the fully connected layer 506. The fused feature representation of the emotion-related feature set may be a representation of the emotion-related feature set, such as R k*D T and can be expressed using equations (8) and (9). Specifically, the vector can be of dimension K * D T The real coordinate space of the

[0083] According to an embodiment, the circuit 104 may be further configured to provide the fused feature representation of the emotion-related feature set as an input to a second fully-connected layer of the output network 310, which may be coupled to the output of the fusion attention network set 308. The circuit 104 may be further configured to predict an emotion label of the corresponding utterance based on the application of the second fully-connected layer of the output network 310. In an embodiment, the output of the first fusion attention network 308A may be passed to the second fusion attention network. This same process may be repeated n times, and the output of the Nth fusion attention network 308N may be passed as an input to the second fully-connected layer (i.e., the output network 310), which may be configured to predict an emotion label of the corresponding utterance.

[0084] Figure 6 illustrates an example architecture of an audio-visual feature extractor of one or more of the feature extractors 304 of Figure 3, according to an embodiment of the present disclosure. Figure 6 is described with reference to elements of Figures 1, 2, 3, 4, and 5. Figure 6 illustrates a diagram 600 of an audio-visual feature extractor 602 of one or more of the feature extractors 304.

[0085] The audiovisual feature extractor 602 may be based on a triplet network. In a triplet network, three input samples may be required. Further shown is an encoder network set 604, which may include a first encoder network 604A, a second encoder network 604B, and a third encoder network 604C. The audiovisual feature extractor 602 may further include a projector set 606. The projector set 606 may include a first projector 606A, a second projector 606B, and a third projector 606C.

[0086] In one embodiment, the input samples 608 associated with the acoustic and visual modalities can be divided into a positive sample set 610A, an anchor sample set 610B, and a negative sample set 610C. Each of the positive sample sets 610A can be similar to the anchor sample set 610B, and each of the positive sample sets 610A can be different from the anchor sample set 610B. The positive sample set 610A can be fed to a first encoder network 604A. The anchor sample set 610B can be fed to a second encoder network 604B. Similarly, the negative sample set 610C can be fed to a third encoder network 604C. Each encoder of the encoder set 604 can have the same architecture with the same number of neurons and associated weights. An example of an encoder can be a ResNet-18 network.

[0087] In an embodiment, the first encoder network 604A can generate a first output. The second encoder network 604B can generate a second output. Similarly, the third encoder network 604C can generate a third output. The first output can be provided as an input to a first projector 606A, the second output can be provided as an input to a second projector 606B, and the third output can be provided as an input to a third projector 606C. Each projector in the projector set 606 can include a fully linear-fully connected layer that can be configured to project an embedding of the encoder network set 604 onto a representation set 612. Specifically, the representation set 612 can include a first representation 612A of the positive sample set 610A, a second representation 612B of the anchor sample set 610B, and a third representation 612C of the negative sample set 610C. The representation set 612 can be mathematically represented using the following equation (13): TIFF2024546432000039.tif13150(13) Where: Z represents the desired expression, N represents the total number of representations, d represents the dimension of each representation.

[0088] The audiovisual feature extractor 602 may be trained using a weighted combination of three loss functions: an adaptive margin triplet loss function, a covariance loss function, and a variance loss function. The training objective of the audiovisual feature extractor 602 may be to decrease the distance between the positive sample set 610A and the anchor sample set 610B, and to increase a second distance between the anchor sample set 610B and the negative sample set 610C. The weighted combination of the three loss functions may be mathematically represented using the following equation (14): TIFF2024546432000040.tif7150(14) Where: λ 1 , λ 2 and λ 3 represents the weighting coefficient, L AMT represents the adaptive margin triplet loss function, L Cov represents the covariance loss function, L Var represents the variance loss function, L FE represents the triplet loss function.

[0089] Traditionally, developers have designed triplet loss functions used to learn good face representations based on a positive sample set 610A, an anchor sample set 610B, and a negative sample set 610C. In their triplet loss functions, developers tend to use fixed margin values ​​that help separate the representations of positive samples from the representations of negative samples. However, when the distance from the positive or negative samples to the anchor is the same, or when the positive samples are slightly closer to the anchor than the negative samples, the triplet loss calculated for such fixed margin values ​​may be zero, and no correction may be observed even though it should still move the positive samples closer to the anchor and the negative samples farther away from the anchor. To solve this problem, an adaptive margin value loss function may be used in the calculation of the triplet loss function. This adaptive margin value loss function may be mathematically expressed using the following equation (15): TIFF2024546432000041.tif13150(15) Where: TIFF2024546432000042.tif7150 represents a Euclidean distance based similarity metric between representations of the positive sample set 610A and representations of the anchor sample set 610B, TIFF2024546432000043.tif7150 represents the Euclidean distance based similarity metric between the representations of the anchor sample set 610B and the representations of the negative sample set 610C, TIFF2024546432000044.tif7150 represents a Euclidean distance based similarity metric between representations of the positive sample set 610A and representations of the negative sample set 610C, m AM represents the adaptive margin.

[0090] In one embodiment, the adaptive margin (m AM ) can be calculated based on the similarity and dissimilarity measures and can be mathematically represented using the following equation (16): TIFF2024546432000045.tif13150(16) Where: TIFF2024546432000046.tif7150 represents the similarity measure, TIFF2024546432000047.tif13150, TIFF2024546432000048.tif13150 represents the similarity measure, The file is TIFF2024546432000049.tif13150.

[0091] In one embodiment, the triple loss function may also include a variance loss function, which may help the audio-visual feature extractor 602 address the mode collapse issue and may be mathematically expressed using the following equation (17): TIFF2024546432000050.tif39150(17) Where: Var(Z) represents the variance obtained from the corresponding representation, TIFF2024546432000051.tif20150, Z k =Z A ,Z p ,Z n and Z p represents a first representation corresponding to the positive sample set 610A, Z a represents a second representation corresponding to the anchor sample set 610B, Z n represents a third representation corresponding to the negative sample set 610C, TIFF2024546432000052.tif7150 represents the average of the corresponding representations.

[0092] In one embodiment, the triple loss function may also include a covariance loss function, which may assist the audio-visual feature extractor 602 in decorrelating different dimensions of the representation and may be mathematically represented using the following equation (18): TIFF2024546432000053.tif39150(18) Where: Cov(Z) denotes the covariance matrix of the corresponding representation, TIFF2024546432000054.tif20150, Z k =Z A ,Z p ,Z n and Z p represents a first representation corresponding to the positive sample set 610A, Z a represents a second representation corresponding to the anchor sample set 610B, Z n represents a third representation corresponding to the negative sample set 610C, TIFF2024546432000055.tif7150 represents the average of the corresponding representations.

[0093] FIG 7 illustrates an example scenario for emotion recognition in multimedia videos using a multimodal fusion based deep neural network, according to an embodiment of the present disclosure. FIG 7 is described with reference to elements of FIG 1, FIG 2, FIG 3, FIG 4, FIG 5 and FIG 6. FIG 7 illustrates a scenario 700. The scenario 700 illustrates a system 102 including the circuit 104 of FIG 1 and the multimodal fusion network 108 of FIG 3. Additionally, multiple multimodal inputs 702 and multiple predicted emotion labels 704 are illustrated.

[0094] The plurality of multimodal inputs 702 may include a first multimodal input 702A, a second multimodal input 702B, a third multimodal input 702C, and an Nth multimodal input 702N. The first multimodal input 702A may relate to a first utterance depicted in one or more videos. The second multimodal input 702B may relate to a second utterance depicted in such videos. Similarly, the third multimodal input 702C may relate to a third utterance depicted in such videos, and the Nth multimodal input 702N may relate to an Nth utterance depicted in such videos. Such utterances may all be part of a conversation (e.g., a two-way conversation). Similar to the plurality of multimodal inputs 702, the plurality of predicted emotion labels 704 may include a first predicted emotion label 704A, a second predicted emotion label 704B, a third predicted emotion label 704C, and an Nth predicted emotion label 704N.

[0095] The circuit 104 may be configured to input a first multi-modal input 702A to one or more feature extractors 110. The first multi-modal input 702A may relate to a first utterance and may include a first modality 706 related to the acoustics of the first utterance, a second modality 708 related to a transcription of the first utterance, and a third modality 710 related to visual aspects of the first utterance.

[0096] The circuit 104 may be further configured to generate an input embedding as an output of one or more feature extractors 110 for the input. The input embedding includes an embedding for each modality of the multi-modal input. More details regarding the input embedding are shown, for example, in FIG. 3.

[0097] The circuit 104 may be further configured to generate an emotion-related feature set based on applying a transformer encoder network to the input embeddings. The emotion-related feature set may include one or more features corresponding to each modality of the multimodal input. After generating the emotion-related feature set, the circuit 104 may be configured to generate a fused feature representation of the emotion-related feature set. In an embodiment, the fused feature representation of the emotion-related feature set may be generated based on applying a fusion attention network to the emotion-related feature set. The circuit 104 may be further configured to output a first predicted emotion label 704A of the first utterance based on applying the output network 116 to the fused feature representation.

[0098] In one embodiment, the above-described operations may be performed for each of the multiple multimodal inputs 702 to predict a corresponding emotion label for the corresponding multimodal input. For example, a first predicted emotion label 704A for a first utterance may be “surprise”. The sentiment associated with the first predicted emotion label 704A may be “positive”. A second predicted emotion label 704B for a second utterance may be “joy” and the corresponding sentiment may be “positive”. A third predicted emotion label 704C for a third utterance may be “neutral” and the corresponding sentiment may be “neutral”. Similarly, an Nth predicted emotion label 704N for an Nth utterance may be “neutral” and the corresponding sentiment may also be “neutral”.

[0099] FIG 8 is a flow chart illustrating an exemplary method for emotion recognition in multimedia videos using multimodal fusion based deep neural networks, according to an embodiment of the present disclosure. FIG 8 is described with reference to elements of FIG 1, FIG 2, FIG 3, FIG 4, FIG 5, FIG 6, and FIG 7. FIG 8 illustrates a flow chart 800. Operation of flow chart 800 may begin at 802 and proceed to 804.

[0100] At 804, multimodal inputs that may be related to speech depicted in one or more videos may be input to one or more feature extractors 110. In at least one embodiment, the circuit 104 may be configured to input multimodal inputs that may be related to speech depicted in one or more videos to one or more feature extractors. More details regarding multimodal inputs are shown, for example, in Figures 1, 3, and 5.

[0101] At 806, an input embedding may be generated as an output of the one or more feature extractors 110 for the input, which may include an embedding for each modality of the multimodal input. In at least one embodiment, the circuit 104 may be configured to generate an input embedding as an output of the one or more feature extractors for the input, which may include an embedding for each modality of the multimodal input. More details regarding the generation of the input embeddings are shown, for example, in FIG. 3.

[0102] At 808, an emotion-related feature set may be generated based on applying the transformer encoder network to the input embeddings, which may include one or more features corresponding to each modality of the multimodal input. In at least one embodiment, the circuit 104 may be configured to generate an emotion-related feature set based on applying the transformer encoder network 112 to the input embeddings, which may include one or more features corresponding to each modality of the multimodal input. More details regarding generating the emotion-related feature set are shown, for example, in FIG. 3.

[0103] At 810, a fused feature representation of the emotion-related Feature set may be generated based on applying the fusion attention network to the emotion-related Feature set. In at least one embodiment, the circuit 104 may be configured to generate a fused feature representation of the emotion-related Feature set based on applying the fusion attention network to the emotion-related Feature set. Details regarding generating the fused feature representation are shown, for example, in FIGS. 3 and 5.

[0104] At 812, an emotion label for the utterance may be predicted based on applying the output network 116 to the fused feature representation. In at least one embodiment, the circuit 104 may be configured to predict an emotion label for the utterance based on applying the output network to the fused feature representation. Control may proceed to an end.

[0105] Experimental data obtained after conducting multiple experiments shows that the disclosed multi-modal fusion network 302 significantly outperforms state-of-the-art methods (i.e., in the range of more than 9% improvement in terms of weighted average F1 score) when performed on known datasets such as the Multimodal Multi-Party Dataset for Emotion Recognition in Conversation (MELD) dataset and the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset.

[0106] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium having stored thereon computer-executable instructions executable by a circuit or machine to operate a system (e.g., system 102) for emotion recognition in multimedia videos using a multimodal fusion-based deep neural network. The computer-executable instructions may cause a machine and / or computer to perform operations including inputting a multimodal input (e.g., multimodal input 124) to one or more feature extractors (e.g., one or more feature extractors 110) of a multimodal fusion network (e.g., multimodal fusion network 108). The multimodal input may relate to speech depicted in one or more videos. The operations may further include generating input embeddings as outputs of the one or more feature extractors for the input. The input embeddings may include embeddings for each modality of the multimodal input. The operations may further include generating an emotion-related feature set based on applying a transformer encoder network (e.g., transformer encoder network 112) of the multimodal fusion network to the input embeddings. The emotion-related feature set includes one or more features corresponding to each modality of the multimodal input. The operations may further include generating a fused feature representation of the emotion-related feature set based on applying a fusion attention network (e.g., fusion attention network 114) of the multimodal fusion network to the emotion-related feature set. The operations may further include predicting an emotion label of the utterance based on applying an output network (e.g., output network 116) of the multimodal fusion network to the fused feature representation.

[0107] Some embodiments of the present disclosure can be found in a system and method for emotion recognition in multimedia videos using multimodal fusion-based deep neural networks. Various embodiments of the present disclosure can provide a system 102 that can include a circuit 104 and a memory 106 configured to store a multimodal fusion network 108 including one or more feature extractors 110, a transformer encoder network 112 coupled to the one or more feature extractors 110, a fusion attention network 114 coupled to the transformer encoder network 112, and an output network 116 coupled to the fusion attention network 114. The circuit 104 can be configured to input a multimodal input 124 to the one or more feature extractors. The multimodal input can relate to speech depicted in the one or more videos. The circuit 104 can be further configured to generate an input embedding as an output of the one or more feature extractors 110 for the input. The input embedding can include an embedding for each modality of the multimodal input. The circuit 104 may be further configured to generate an emotion-related feature set based on applying the transformer encoder network 112 to the input embeddings. The emotion-related feature set includes one or more features corresponding to each modality of the multimodal input. The circuit 104 may be further configured to generate a fused feature representation of the emotion-related feature set based on applying the fusion attention network 114 to the emotion-related feature set. The circuit 104 may be further configured to predict an emotion label of the utterance based on applying the output network 310 to the fused feature representation.

[0108] According to an embodiment, the multimodal input 124 includes multilingual speech and a transcription of the multilingual speech in a first language that can be matched to one or more feature extractors. According to an embodiment, the multimodal input includes speech in a second language that can be different from the first language that can be matched to the one or more feature extractors 110, and the multimodal input includes a transcription of the speech in the first language that can be matched to the one or more feature extractors 110. According to an embodiment, the multimodal input includes a first modality 314A related to the acoustics of the speech, a second modality 314B related to the transcription of the speech, and a third modality 314C related to visual aspects of the speech.

[0109] According to an embodiment, the one or more feature extractors may include an acoustic feature extractor 304B and an audio-visual feature extractor 304C, and the circuit 104 may be further configured to generate a first embedding of the input embedding based on applying one of the audio-visual feature extractor 304C or the acoustic feature extractor 304B to acoustic information of the speech included in the multimodal input 124.

[0110] According to an embodiment, the one or more feature extractors may include a text feature extractor 304C, and the circuit 104 may be further configured to generate a second embedding of the input embedding based on applying the text feature extractor 304C to the transcription of acoustic information associated with the utterance and to a transcription of a different utterance that precedes or follows the utterance in time.

[0111] According to an embodiment, the one or more feature extractors may include a visual feature extractor 304D and an audio-visual feature extractor 304C, and the circuit 104 may be further configured to generate a third embedding of the input embedding based on applying one of the audio-visual feature extractor 304C or the visual feature extractor 304D to facial information of one or more speaking characters in frames of the one or more videos and scene information associated with the frames. The frames may correspond to durations of speech in the one or more videos.

[0112] According to an embodiment, the circuit 104 may be configured to input one or more frames of the video corresponding to a duration of the utterance to the visual feature extractor 304D. The circuit 104 may be further configured to detect one or more faces in each of the received frames based on applying the face detection model 210 of the visual feature extractor 304D to each of the received frames. The circuit 104 may be further configured to generate one or more bounding boxes that contain the detected one or more faces. The circuit 104 may be further configured to normalize an area associated with each of the one or more bounding boxes by applying the visual feature extractor 304D. The circuit 104 may be further configured to generate a third embedding of the input embedding as an output of the visual feature extractor 304D based on the detected one or more faces and the normalization.

[0113] According to an embodiment, the transformer encoder network 306 may include a first stack 316 of transformer encoders for a first modality 314A of the multimodal input, a second stack 318 of transformer encoders for a second modality 314B of the multimodal input, and a third stack 320 of transformer encoders for a third modality 314C of the multimodal input.

[0114] According to an embodiment, the system 102 may further include a skip connection 322 between each pair of adjacent transformer encoders in the transformer encoder network 306 .

[0115] According to an embodiment, the circuit 104 may be configured to receive one or more videos. The circuit 104 may be further configured to apply a scene detection model 212 to the received one or more videos. The circuit 104 may be further configured to extract a plurality of scenes from the one or more videos based on the application of the scene detection model. The circuit 104 may be further configured to apply a single boundary detection model 214 to each of the extracted plurality of scenes. The circuit 104 may be further configured to detect a plurality of speeches 312 in the extracted plurality of scenes based on the application of the single boundary detection model 214. The circuit 104 may be further configured to prepare a set of multimodal inputs based on the detection. The multimodal inputs that may be input to the one or more feature extractors 304 may be part of the prepared set of multimodal inputs.

[0116] According to an embodiment, each of the set of fusion attention networks 308 may include one or more multi-head attention layers and a first fully connected layer, where the input of the first fully connected layer may be connected to the output of the one or more multi-head attention layers.

[0117] According to an embodiment, the circuitry 104 may be further configured to apply one or more multi-head attention layers to the emotion-related feature set to determine a feature-to-feature mapping within the emotion-related feature set, and to concatenate the emotion-related feature set to a latent representation of the emotion-related feature set based on the feature-to-feature mapping.

[0118] According to an embodiment, a fused feature representation of the emotion-related feature set may be further generated based on applying a first fully connected layer to the latent representation.

[0119] According to one embodiment, the output network 310 may include a second fully connected layer coupled to the output of the fusion attention network set 308.

[0120] The present disclosure can be realized in hardware or in a combination of hardware and software. The present disclosure can be realized in a centralized manner in at least one computer system, or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other device adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system that includes a computer program that, when loaded and executed, can control a computer system to perform the methods described herein. The present disclosure can be realized in hardware, including parts of an integrated circuit that also perform other functions.

[0121] The present disclosure may also be embodied in a computer program product, which includes all features enabling the implementation of the methods described herein and which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system capable of processing information to execute a particular function, either directly, or after a) conversion into another language, code or notation, b) reproduction in a different content form, or both.

[0122] While the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the present disclosure. Many modifications can also be made to adapt a particular situation or material to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is not intended that the present disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims. [Explanation of symbols]

[0123] 210 Face Detection Model 302 Multimodal Fusion Network 304A Audio-visual feature extractor 304D Visual Feature Extractor 402 System 404 Frame 406 First Face 408 Second Face 408 First Bounding Box 412 Secondary Bounding Box< / s> < / s> < / s>

Claims

1. 1. A system comprising: a memory configured to store a multimodal fusion network including one or more feature extractors, the feature extractors including an audio-visual feature extractor and a visual feature extractor, a transformer encoder network coupled to the one or more feature extractors, a fusion attention network coupled to the transformer encoder network, and an output network coupled to the fusion attention network; inputting multimodal input related to speech depicted in one or more videos into the one or more feature extractors; generating an input embedding as an output of the one or more feature extractors for the input, the input embedding including an embedding for each modality of the multimodal input; generating an emotion-related feature set, the emotion-related feature set including one or more features corresponding to each modality of the multimodal input, based on applying the transformer encoder network to the input embeddings; generating a fused feature representation of the emotion-related feature set based on applying the fusion attention network to the emotion-related feature set; predicting an emotion label for the utterance based on applying the output network to the fused feature representation; A circuit configured as follows: Equipped with the circuitry is further configured to generate a third embedding of the input embedding based on applying one of the audio-visual feature extractor or the visual feature extractor to facial information of one or more speaking characters in frames of the one or more videos and scene information associated with the frames; a frame corresponding to a duration of the utterance in the one or more videos; The circuit comprises: inputting the one or more frames of the video corresponding to the duration of the utterance into the audio-visual feature extractor or the visual feature extractor; detecting one or more faces in each of the received frames based on applying a face detection model of the audiovisual feature extractor or the visual feature extractor to each of the received frames; generating one or more bounding boxes containing the detected face or faces; normalizing an area associated with each of the one or more bounding boxes by applying one of the audiovisual feature extractor or the visual feature extractor; generating a third embedding of the input embedding as an output of the audio-visual feature extractor or the visual feature extractor based on the detected one or more faces and the normalization; The system further configured as follows.

2. the multimodal input includes multilingual speech and a transcription of the multilingual speech in a first language compatible with the one or more feature extractors; The system of claim 1 .

3. the multimodal input includes speech in a second language different from a first language compatible with the one or more feature extractors, and the multimodal input includes a transcription of the speech in the first language compatible with the one or more feature extractors; The system of claim 1 .

4. the multimodal input includes a first modality related to the acoustics of the utterance, a second modality related to a transcription of the utterance, and a third modality related to a visual aspect of the utterance; The system of claim 1 .

5. the one or more feature extractors include an audio-visual feature extractor and an acoustic feature extractor, and the circuitry is further configured to generate a first embedding of the input embedding based on applying one of the audio-visual feature extractor or the acoustic feature extractor to acoustic information of the utterance included in the multimodal input. The system of claim 1 .

6. the one or more feature extractors include a text feature extractor, and the circuitry is further configured to generate a second embedding of the input embedding based on applying the text feature extractor to a transcription of acoustic information associated with the utterance and to a transcription of a different utterance that temporally precedes or follows the utterance. The system of claim 1 .

7. the transformer encoder network includes a first stack of transformer encoders for a first modality of the multimodal input, a second stack of transformer encoders for a second modality of the multimodal input, and a third stack of transformer encoders for a third modality of the multimodal input; The system of claim 1 .

8. further comprising a skip connection between each pair of adjacent transform encoders in the transform encoder network. The system of claim 1 .

9. The circuit comprises: receiving the one or more videos; applying a scene detection model to the received one or more videos; extracting scenes from the one or more videos based on application of the scene detection model; applying a single boundary detection model to each of the extracted scenes; detecting utterances in the extracted scenes based on application of the single boundary detection model; preparing a set of multimodal inputs based on said detection; wherein the multimodal input input to the one or more feature extractors is part of the prepared set of multimodal inputs. The system of claim 1 .

10. the fusion attention network includes one or more multi-head attention layers and a first fully connected layer; the input of the first fully connected layer is connected to the output of the one or more multi-head attention layers; The system of claim 1 .

11. The circuitry applies one or more multi-head attention layers to the emotion-related feature set; determining a mapping between features in the emotion-related feature set; concatenating the emotion-related feature set to a latent representation of the emotion-related feature set based on the feature-to-feature mapping; The system of claim 10 further configured to:

12. the fused feature representation of the emotion-related feature set is generated further based on applying the first fully connected layer to the latent representation. The system of claim 11.

13. the output network includes a second fully connected layer coupled to the output of the fusion attention network; The system of claim 1 .

14. 1. A system including a memory that stores a multimodal fusion network, inputting multimodal inputs related to speech depicted in one or more videos to one or more feature extractors of the multimodal fusion network, the feature extractors including an audiovisual feature extractor and a visual feature extractor; generating an input embedding as an output of the one or more feature extractors for the input, the embedding including an embedding for each modality of the multimodal input; generating an emotion-related feature set, the emotion-related feature set including one or more features corresponding to each modality of the multimodal input, based on applying a transformer encoder network of the multimodal fusion network to the input embeddings; and generating a fused feature representation of the emotion-related feature set based on applying a fusion attention network of the multimodal fusion network to the emotion-related feature set; predicting an emotion label of the utterance based on applying an output network of the multimodal fusion network to the fused feature representation; and Including, generating a third embedding of the input embedding based on applying one of the audiovisual feature extractor or the visual feature extractor to facial information of one or more speaking characters in frames of the one or more videos and scene information associated with the frames; a frame corresponding to a duration of the utterance in the one or more videos; inputting the one or more frames of the video corresponding to the duration of the utterance into the audio-visual feature extractor or the visual feature extractor; detecting one or more faces in each of the received frames based on applying a face detection model of the audiovisual feature extractor or the visual feature extractor to each of the received frames; generating one or more bounding boxes containing the detected face or faces; normalizing an area associated with each of the one or more bounding boxes by applying one of the audiovisual feature extractor or the visual feature extractor; generating a third embedding of the input embedding as an output of the audiovisual feature extractor or the visual feature extractor based on the detected one or more faces and the normalization; The method further comprising:

15. the multimodal input includes a first modality related to the acoustics of the utterance, a second modality related to a transcription of the utterance, and a third modality related to a visual aspect of the utterance; 15. The method of claim 14.

16. the fusion attention network includes one or more multi-head attention layers and a first fully connected layer; the input of the first fully connected layer is connected to the output of the one or more multi-head attention layers; 15. The method of claim 14.

17. applying one or more multi-head attention layers to the emotion-related feature set; determining a mapping between features in the emotion-related feature set; concatenating the emotion-related feature set to a latent representation of the emotion-related feature set based on the feature-to-feature mapping; wherein the fused feature representation of the emotion-related feature set is generated further based on applying the first fully connected layer to the latent representation.

17. The method of claim 16.

18. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a circuit of a system, cause the circuit to perform operations, the operations including: inputting multimodal inputs related to speech depicted in one or more videos to one or more feature extractors of a multimodal fusion network; generating an input embedding as an output of the one or more feature extractors for the input, the embedding including an embedding for each modality of the multimodal input; generating an emotion-related feature set, the emotion-related feature set including one or more features corresponding to each modality of the multimodal input, based on applying a transformer encoder network of the multimodal fusion network to the input embeddings; and generating a fused feature representation of the emotion-related feature set based on applying a fusion attention network of the multimodal fusion network to the emotion-related feature set; predicting an emotion label of the utterance based on applying an output network of the multimodal fusion network to the fused feature representation; and Including, The operation is generating a third embedding of the input embedding based on applying one of the audiovisual feature extractor or the visual feature extractor to facial information of one or more speaking characters in frames of the one or more videos and scene information associated with the frames; a frame corresponding to a duration of the utterance in the one or more videos; The operation is inputting the one or more frames of the video corresponding to the duration of the utterance into the audio-visual feature extractor or the visual feature extractor; detecting one or more faces in each of the received frames based on applying a face detection model of the audiovisual feature extractor or the visual feature extractor to each of the received frames; generating one or more bounding boxes containing the detected face or faces; normalizing an area associated with each of the one or more bounding boxes by applying one of the audiovisual feature extractor or the visual feature extractor; generating a third embedding of the input embedding as an output of the audiovisual feature extractor or the visual feature extractor based on the detected one or more faces and the normalization; 10. A non-transitory computer-readable medium, further comprising:

Images