Emotion recognition in multimedia video using a multimodal fusion-based deep neural network

Abstract

A landmark detection system and method using emotion recognition in multimedia videos using a multimodal fusion based deep neural network is provided. The system includes a circuit and a memory configured to store a multimodal fusion network including one or more feature extractors, a transformer encoder network, a fusion attention network, and an output network coupled to the fusion attention network. The system inputs a multimodal input to the one or more feature extractors. The multimodal input is related to an utterance depicted in the one or more videos. The system generates an input embedding as an output of the one or more feature extractors for the input, and further generates an emotion-related feature set based on the input embedding. The system further generates a fused feature representation of the emotion-related feature set, and predicts an emotion label of the utterance based on the fused feature representation.

Description

[Technical Field]

[0001] [Incorporation by cross-referencing / citation with related applications] This application claims priority to U.S. Provisional Patent Application No. 63 / 263,961, filed on 12 November 2021, and to U.S. Patent Application No. 17 / 941,787, filed with the U.S. Patent and Trademark Office on 9 September 2022, the contents of which are incorporated herein by reference in their entirety.

[0002] Various embodiments of this disclosure relate to neural networks and emotion recognition. Specifically, various embodiments of this disclosure relate to systems and methods for emotion recognition in multimedia video using multimodal fusion-based deep neural networks. [Background technology]

[0003] Advances in computer vision and artificial intelligence have led to the development of various types of neural networks (or models) that can be used for different applications, such as emotion recognition in conversation. Emotion recognition is typically used to predict the emotional state of speakers from (single / plural) conversations depicted in multimedia video (e.g., movies, web series, and news). Emotion recognition is crucial in the development of empathetic human-machine systems. In the case of conversation, conventional approaches to emotion recognition rely almost entirely on text transcripts. Inaccuracies in the transcripts can affect the accuracy of predictions (e.g., emotion labels). Many state-of-the-art techniques for emotion recognition ignore the vast amount of information present in the visual and acoustic signals associated with the conversation. [Overview of the project] [Problems that the invention aims to solve]

[0004] Those skilled in the art will appreciate the limitations and disadvantages of conventional and customary techniques by comparing the described system with some aspects of the present disclosure illustrated while referring to the drawings in the remainder of this application. **Means for Solving the Problem**

[0005] Provided are a system and method for emotion recognition in multimedia video using a multimodal fusion-based deep neural network, which is illustrated and / or described in relation to at least substantially one figure and further shown more fully in the claims.

[0006] These and other features and advantages of the present disclosure will be understood by considering the following detailed description of the present disclosure while referring to the accompanying drawings, in which like elements are designated by like reference numerals throughout. **Brief Description of the Drawings**

[0007] [Figure 1] FIG. 1 is a diagram showing a network environment for emotion recognition in multimedia video using a multimodal fusion-based deep neural network according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram of an exemplary system for emotion recognition in multimedia video using a multimodal fusion-based deep neural network according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram showing an exemplary architecture of the multimodal fusion network of FIG. 1 according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram showing an exemplary visual feature extractor of the multimodal fusion attention network of FIG. 3 according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a diagram showing an exemplary architecture of a certain fusion attention network among the fusion attention network sets of FIG. 3 according to an embodiment of the present disclosure. [Figure 6] FIG. is a diagram showing an exemplary architecture of an audiovisual feature extractor of one or more feature extractors according to an embodiment of the present disclosure. [Figure 7] FIG. is a diagram showing an exemplary scenario for emotion recognition in multimedia video using a multimodal fusion-based deep neural network according to an embodiment of the present disclosure. [Figure 8] FIG. is a flowchart showing an exemplary method for emotion recognition in multimedia video using a multimodal fusion-based deep neural network according to an embodiment of the present disclosure.

MODE FOR CARRYING OUT THE INVENTION

[0008] The system and method for emotion recognition in multimedia video using a multimodal fusion-based deep neural network disclosed herein can be found in the implementation described below. The system includes circuitry and memory, the memory being configured to store a multimodal fusion network comprising one or more feature extractors, a transformer encoder network coupled to one or more feature extractors, a fusion attention network coupled to the transformer encoder network, and an output network coupled to the fusion attention network. The system can take a multimodal input as input to one or more feature extractors. The multimodal input may relate to utterances depicted in one or more videos (such as movies). The system can generate input embeddings as the output of one or more feature extractors to the input. The input embeddings may include embeddings for each modality of the multimodal input. The system can further generate emotion-related feature sets based on applying transformer encoder networks to the input embeddings. The emotion-related feature sets include one or more features corresponding to each modality of the multimodal input. The system can further generate fused-feature representations of the emotion-related feature set by applying a fusion attention network to the emotion-related feature set. The system can then predict the emotion label (such as anger, neutrality, joy, or sadness) of an utterance by applying an output network to the fused-feature representation.

[0009] Emotions can be described as invisible mental states that may be related to a person's thoughts and feelings. Without physiological indications, emotions can only be detected through human behavior such as written speech, visual gestures, and acoustic signals. Speech-based emotion recognition attempts to recognize a person's emotions in a conversation by relying on written, visual, and acoustic cues. Currently, speech-based emotion recognition has become an essential task in the context of multimedia content analysis and moderation (such as video), helping to understand the nature of user-content interaction. Speech-based emotion recognition has other important applications in many other tasks, including AI interviews, personalized dialogue systems, opinion mining across chat histories, and understanding user perception of content within social media platforms.

[0010] Current state-of-the-art emotion recognition methods frame the task of recognizing emotions in conversation as a purely text-based task. Specifically, these methods determine the emotional state of a subject based on text data related to that subject. This text data can correspond to a transcript of the subject's spoken words. However, the vast amount of information present in the acoustic and visual modalities of multimedia content is not taken into account in determining the subject's emotional state.

[0011] This disclosure provides a neural network architecture for detecting a subject's emotional state using at least three different modalities related to speech (acoustic modality, textual modality, and visual modality). Experimental data suggests that correctly fusing the three modalities can improve the quality and robustness of current state-of-the-art systems. The disclosed system can detect a subject's emotional state by taking each modality contributing to emotion prediction as input. The disclosed method can be further generalized compared to current state-of-the-art methods.

[0012] This disclosure also provides an acoustic-visual feature extractor that can be designed based on a triplet network to leverage the importance of the triplet loss function. The acoustic-visual feature extractor is trained on triplet loss functions including an adaptive margin triplet loss function, a covariance loss function, and a variance loss function.

[0013] Figure 1 shows a network environment for emotion recognition in multimedia video using a multimodal fusion-based deep neural network according to an embodiment of the present disclosure. Figure 1 shows a diagram of the network environment 100. The network environment 100 includes a system 102. The system 102 includes a circuit 104 and a memory 106. The memory may include a multimodal fusion network 108. The multimodal fusion network 108 may include one or more feature extractors 110, a transformer encoder network 112, a fusion attention network 114, and an output network 116. Figure 1 further shows a display device 118, a server 120, and a communication network 122. Figure 1 also shows a multimodal input 124 and a predicted emotion label 126 displayed on the display device 118.

[0014] Circuit 104 may include preferred logic, circuits, and interfaces that can be configured to execute program instructions related to different operations that the system 102 should perform. Circuit 104 can be implemented based on several processor technologies well known in the art. Examples of processor technologies include, but are not limited to, central processing units (CPUs), x86-based processors, reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, composite instruction set computing (CISC) processors, graphics processing units (GPUs), coprocessors (such as inference accelerators or artificial intelligence (AI) accelerators), and / or combinations thereof.

[0015] Memory 106 may include preferred logic, circuitry, and / or interfaces that can be configured to store program instructions that the circuit 104 can execute. Memory 106 may also store the multimodal fusion network 108. In at least one embodiment, memory 106 may also store input data for the multimodal fusion network 108, intermediate results obtained using multimodal input embeddings, and (single / multiple) sentiment labels predicted by the multimodal fusion network 108. Examples of implementations of memory 106 include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drives (HDDs), solid-state drives (SSDs), CPU caches, and / or secure digital (SD) cards.

[0016] The multimodal fusion network 108 can be a system of computational networks or artificial neurons arranged in multiple layers. The multimodal fusion network 108 can be trained to predict emotion labels (e.g., emotion labels 126) of utterances depicted in one or more videos (i.e., emotion recognition in conversation).

[0017] The multimodal fusion network 108 includes one or more feature extractors 110, a transformer encoder network 112 coupled to the one or more feature extractors 110, a fusion attention network 114 coupled to the transformer encoder network 112, and an output network 116 coupled to the fusion attention network 114. The operation of the multimodal fusion network 108 can be divided into two stages. In the first stage (i.e., utterance level), features can be extracted individually at the utterance level. Then, in the second stage (i.e., dialogue level), the network can learn to predict the sentiment label of each utterance by using contextual information from the dialogue. An utterance can correspond to a short oral segment spoken by one of the parties in a multi-person conversation and may have a transcript, video clip, and audio segment. A dialogue may include multiple such utterances in the order in which they occurred in time.

[0018] Each of the one or more feature extractors 110 may be configured to perform one or more operations to generate input embeddings for each modality of the multimodal input (e.g., multimodal input 124). Each encoder in the transformer encoder network 112 may be configured to perform one or more operations to generate a set of sentiment-related features. The fusion attention network 114 may be configured to generate a fused feature representation of the sentiment-related feature set. Specifically, the fusion attention network 114 may be configured to generate a fused feature representation based on applying one or more multi-head attention operations to the sentiment-related feature set. The output network 116 may be configured to predict sentiment labels 126 of utterances associated with the multimodal input 124. The output network 116 can predict sentiment labels 126 based on the fused feature representation. The output network 116 may include a fully connected layer that can be configured to predict sentiment labels 126. Details of the output network 116 are shown, for example, in Figure 3.

[0019] Each of the one or more feature extractors 110, each encoder in the transformer encoder network 112, the fusion attention network 114, and the output network 116 can be a system of neural networks or artificial neurons arranged in multiple layers. The multiple layers of the neural network can include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers can include one or more nodes (i.e., artificial neurons). The outputs of all nodes in the input layer can be coupled to at least one node in one or more hidden layers. Similarly, the input of each hidden layer can be coupled to the output of at least one node in another layer of the neural network. The output of each hidden layer can be coupled to the input of at least one node in another layer of the neural network. The nodes in the final layer can receive input from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer can be determined from the hyperparameters of the neural network. Such hyperparameters can be set before or after training the neural network on the training dataset. In the multimodal fusion network 108, the training dataset can include a set of multimodal inputs and corresponding sentiment labels as ground truth values. Each multimodal input can include at least one of the following: audio of an utterance, one or more frames in a scene in which one or more characters produce the utterance, and a transcript of the audio.

[0020] Each node in a neural network can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with a set of parameters that can be adjusted during network training. These parameters may include, for example, weight parameters and regularization parameters. Each node can use the mathematical function to compute an output based on one or more inputs from nodes in other (single or multiple) layers of the neural network (e.g., previous (single or multiple) layers). All or some nodes in a neural network can correspond to the same or different mathematical functions.

[0021] In training a neural network, one or more parameters of each node in the neural network can be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on the neural network's loss function. This process can be repeated for the same or different inputs until the minimum value of the loss function is achieved and the training error is minimized. Several training methods are known in this field, including gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristics.

[0022] Each of the one or more feature extractors 110, each encoder in the transformer encoder network 112, the fusion attention network 114, and the output network 116 may include electronic data that can be implemented, for example, as software components of an application executable on system 102. Each of the one or more feature extractors 110, each encoder in the transformer encoder network 112, the fusion attention network 114, and the output network 116 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device such as circuit 104. Each encoder in the transformer encoder network 112, the fusion attention network 114, and the output network 116 may include code and routines configured to enable a computer device such as circuit 104 to perform one or more operations. For example, each of the one or more feature extractors 110 may perform one or more operations to generate input embeddings for each modality of the received multimodal input 124. Each encoder in the transformer encoder network 112 may perform one or more operations to generate a set of sentiment-related features. In addition to or instead of the above, each of the one or more feature extractors 110, each encoder of the transformer encoder network 112, the fusion attention network 114, and the output network 116 may also be implemented using hardware including a processor, a microprocessor (for example, one or more operations performed or controlled thereunder), a tensor processing unit (TPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, each of the one or more feature extractors 110, each encoder of the transformer encoder network 112, the fusion attention network 114, and the output network 116 may also be implemented using a combination of hardware and software.

[0023] In one embodiment, each encoder in the transformer encoder network 112 can be configured to receive input embeddings for each modality in parallel (i.e., simultaneously) and to simultaneously provide emotion-related feature sets as outputs. Each encoder may include, but is not limited to, a multi-head attention layer and a feedforward neural network.

[0024] In one embodiment, a fusion attention network 114 can be used to incorporate visual and acoustic modalities into a text modality related to speech. The fusion attention network 114 may include one or more multi-head attention layers and a first fully connected layer. Details of the fusion attention network are shown, for example, in Figure 5.

[0025] The display device 118 may include suitable logic, circuitry, and interfaces that can be configured to display emotion labels 126 of utterances associated with the multimodal input 124. In one embodiment, the display device 118 may be configured to display the multimodal input 124 and emotion labels 126 corresponding to the utterance level portion of the multimodal input 124. The display device 118 can be implemented through a number of known technologies, including, but not limited to, liquid crystal display (LCD) displays, light-emitting diode (LED) displays, plasma displays, or organic LED (OLED) display technologies, or at least one of other display devices. According to one embodiment, the display device 118 may mean a display screen for a head-mounted device (HMD), a smart glasses device, a see-through display, a projected display, an electrochromic display, or a transparent display.

[0026] In another embodiment, the display device 118 may include suitable logic, circuits, interfaces and / or code that can implement the multimodal fusion network 108 as part of a software program or service (such as an application programming interface (API)-based service) that can run on the display device 118. The multimodal fusion network 108 can be implemented on the display device 118 after training of the multimodal fusion network 108 on system 102 has been completed. Examples of the display device 118 include, but are not limited to, computer devices, mainframe machines, servers, computer workstations, smartphones, mobile phones, game devices, wearable displays, consumer electronic (CE) devices, and / or any other device having image processing capabilities.

[0027] Server 120 may include suitable logic, circuitry, and interfaces, and / or code, which can be configured to store one or more videos for the purpose of emotion recognition and other operations such as media streaming. Server 120 may also be configured to store predictable emotion labels 126 or speech level portions of videos. Server 120 can be implemented as a cloud server and can perform operations through web applications, cloud applications, HTTP requests, repository operations, and file transfers. Other implementation examples of Server 120 include, but are not limited to, database servers, file servers, web servers, application servers, mainframe servers, or cloud computing servers.

[0028] In at least one embodiment, the server 120 can be implemented as multiple distributed cloud-based resources by using some techniques well known to those skilled in the art. Those skilled in the art will understand that the scope of this disclosure may not be limited to implementations of the server 120 and system 102 as two independent entities. In some embodiments, the functionality of the server 120 can be incorporated into system 102, either entirely or at least partially, without departing from the scope of this disclosure.

[0029] The communication network 122 may include a communication medium that enables the system 102, the display device 118, and the server 120 to communicate with each other. The communication network 122 may include either a wired connection or a wireless connection. Examples of the communication network 122 include, but are not limited to, the internet, a cloud network, a cellular or wireless mobile network (such as Long Term Evolution or 5G New Wireless), a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices within the network environment 100 may be configured to connect to the communication network 122 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, Multihop Communication, Wireless Access Point (AP), Device-to-Device Communication, Cellular Communication Protocol, and Bluetooth (BT) Communication Protocol.

[0030] During operation, the circuit 104 can be configured to input a multimodal input 124 to one or more feature extractors 110. The multimodal input may relate to utterances depicted in one or more videos. For example, the multimodal input may include a first modality related to the acoustics of the utterance, a second modality related to the transcription of the utterance, and a third modality related to the visual aspects of the utterance. In one embodiment, the multimodal input may further include a fourth modality which may relate to one or more biological parameters of the subject (i.e., the speaker) related to the corresponding utterance.

[0031] Circuit 104 can be configured to generate input embeddings as the output of one or more feature extractors 110 for the input. The input embeddings may include embeddings for each modality of the multimodal input 124. In one embodiment, the input embeddings may correspond to features of the corresponding modalities. Specifically, the input embeddings may be called low-dimensional learned continuous vector representations of discrete variables. Based on the generation of input embeddings, circuit 104 can be further configured to generate sentiment-related feature sets based on applying a transformer encoder network 112 to the input embeddings. In one embodiment, the transformer encoder network 112 can be applied to the input embeddings to learn the context of the corresponding utterances for each modality. The sentiment-related feature sets may include one or more features corresponding to each modality of the multimodal input 124. Details regarding sentiment-related feature sets are shown, for example, in Figure 3.

[0032] To map each modality into the corresponding text vector space, the circuit can be configured to apply the fusion attention network 114 to the emotion-related feature set. In another embodiment, the fusion attention network 114 can be used to incorporate visual and acoustic information relevant to the dialogue. Specifically, the circuit 104 can be further configured to generate a fused feature representation of the emotion-related feature set based on applying the fusion attention network 114 to the emotion-related feature set. Details regarding the fused feature representation are shown, for example, in Figure 3.

[0033] After the generation of the fused feature representation, the circuit 104 can be configured to predict sentiment labels 126 for utterances associated with the multimodal input 124. The sentiment labels 126 can be predicted based on applying the output network 116 to the fused feature representation. In one embodiment, the circuit 104 can be further configured to control the display device 118 to render the predicted sentiment labels 126 onto the display device 118. Details regarding the sentiment labels 126 and the prediction of sentiment labels 126 are shown, for example, in Figure 3.

[0034] Figure 2 is an exemplary block diagram of a system for emotion recognition in multimedia video using a multimodal fusion-based deep neural network according to an embodiment of the present disclosure. The description of Figure 2 will be made in relation to the elements of Figure 1. Figure 2 shows a block diagram 200 of the system 102 of Figure 1. The system includes a circuit 104, memory 106, a multimodal fusion network 108, input / output (I / O) devices 202, a network interface 204, an inference accelerator 206, a translator model 208, a face detection model 210, a scene detection model 212, and a single boundary detection model 214.

[0035] The I / O device 202 may include suitable logic, circuitry, and / or interfaces that can be configured to receive one or more user inputs and / or render information (such as the predicted sentiment label 126) generated by the system 102. The I / O device 202 may include various input / output devices that can be configured to communicate with different operating components of the system 102. Examples of the I / O device 202 include, but are not limited to, a touch screen, keyboard, mouse, joystick, microphone, and display devices (such as the display device 118).

[0036] The network interface 204 may include suitable logic, circuitry, interfaces, and / or code that can be configured to establish communication between the system 102, the display device 118, and the server 120 via the communication network 122. The network interface 204 may be configured to implement known technologies that support wired or wireless communication. The network interface 204 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0037] The network interface 204 can be configured to communicate with networks such as the Internet, intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (WLANs), personal area networks, and / or metropolitan area networks (MANs) via offline and online wireless communication. Wireless communication can use any of several communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), LTE, 5G New Wireless, Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (WiFi) (such as IEEE 802.11, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and / or any other IEEE 802.11 protocol), Voice over Internet Protocol (VoIP), Wi-MAX, Internet of Things (IoT) technology, Machine Communication (MTC) technology, and protocols for email, instant messaging, and / or Short Message Service (SMS).

[0038] The inference accelerator 206 may include suitable logic, circuitry, interfaces, and / or code that can be configured to act as a coprocessor for circuit 104 to accelerate computations related to the operation of the multimodal fusion network 108. For example, the inference accelerator 206 can accelerate computations on system 102 so that sentiment labels 126 can be predicted in less time than would normally occur without the inference accelerator 206. The inference accelerator 206 can implement various acceleration techniques, such as parallelizing the operation of some or all of the one or more feature extractors 110, transformer encoder network 112, fusion attention network 114, and output network 116. The inference accelerator 206 can be implemented as software, hardware, or a combination thereof. Implementation examples of the inference accelerator 206 include, but are not limited to, GPUs, tensor processing units (TPUs), neuromorphic chips, vision processing units (VPUs), field-programmable gate arrays (FPGAs), reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, composite instruction set computing (CISC) processors, microcontrollers, and / or combinations thereof.

[0039] The translator model 208 may include suitable logic, circuitry, interfaces, and / or code that can be configured to translate speech in a second language into a first language (or vice versa). In one embodiment, the translator model 208 may be configured to translate a copy of speech in a first language into a second language. Examples of the translator model 208 include, but are not limited to, artificial neural networks (ANNs), deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), gated recurrent unit (GRU) based RNNs, CNN-recurrent neural network (CNN-RNNs), long short-term memory (LSTM) network based RNNs, LSTM+ANNs, and / or combinations of such networks.

[0040] The face detection model 210 may include suitable logic, circuitry, interfaces, and / or code that can be configured to detect one or more faces in an image (or frame). The face detection model 210 can detect one or more faces in an image using one or more face detection techniques. Detailed implementations of one or more face detection techniques are well known to those skilled in the art, and therefore, for the sake of brevity, a detailed description of the one or more face detection techniques described above is omitted from this disclosure. Examples of the face detection model 210 include, but are not limited to, convolutional neural networks (CNNs), R-CNNs, Faster R-CNNs, (You Only Look Once) YOLO networks, and / or combinations of such networks.

[0041] The scene detection model 212 may include suitable logic, circuitry, interfaces, and / or code that can be configured to extract multiple scenes from one or more videos. In one embodiment, multiple scenes may be extracted based on background pixel information (e.g., changes in background pixel values). Examples of the scene detection model 212 include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), artificial neural networks (ANNs), and / or combinations of such networks.

[0042] The single boundary detection model 214 may include preferred logic, circuitry, interfaces, and / or code that can be configured to detect multiple utterances within a scene. In one embodiment, the single boundary detection model 214 may include a VGG-16 convolutional neural network (CNN) followed by multiple convolutional layers. The VGG-16 can be used for feature extraction, and the convolutional layers can be used for object detection. The single boundary detection model 214 may further extract multiple scenes of the video based on object detection. In one embodiment, the single boundary detection model 214 may extract multiple scenes of the video based on background pixel information (e.g., changes in background pixel values).

[0043] Figure 3 shows an exemplary architecture of the multimodal fusion network of Figure 1 according to an embodiment of the present disclosure. The description of Figure 3 will be made in relation to the elements of Figures 1 and 2. Figure 3 shows Figure 300 of a multimodal fusion network 302 which may be an exemplary implementation of the multimodal fusion network 108 of Figure 1.

[0044] The multimodal fusion network 302 may include one or more feature extractors 304, a transformer encoder network 306, a fusion attention network set 308, and an output network 310. The transformer encoder network 306 can be coupled to one or more feature extractors 304, the first fusion attention network 308A of the fusion attention network set 308 can be coupled to the transformer encoder network 306, and the output network 310 can be coupled to the nth fusion attention network 308N of the fusion attention network set 308. According to one embodiment, there may be a single fusion attention network (e.g., the first fusion attention network 308A) that can be coupled to the transformer encoder network 306 and the output network 310. Figure 3 further shows block-style representations of multiple utterances 312 corresponding to conversations depicted in a video or one or more videos.

[0045] At some point, circuit 104 can receive one or more videos that depict a conversation between multiple human speakers or characters (e.g., a dyadic conversation). It needs to predict a sentiment label for each utterance in such a conversation. The operation of predicting sentiment labels is described here.

[0046] Circuit 104 can be configured to apply the scene detection model 212 to one or more video frames after reception. As illustrated in Figure 2, the scene detection model 212 can be trained to extract multiple scenes (or multiple dialogues) from each video. Circuit 104 can process video frames corresponding to multiple scenes for automatic utterance detection. As an example, but not an exhaustive one, a single boundary detection model 214 can be applied to a frame to detect multiple objects in each of such frames. Based on this detection, Circuit 104 can further detect subsets of frames corresponding to multiple utterances 312.

[0047] Circuit 104 can extract audio portions from one or more videos. Each of these audio portions may include speech sounds corresponding to utterances (as part of a conversation or dialogue). Similarly, circuit 104 can generate transcripts of such audio portions using preferred speech-to-text (STT) techniques.

[0048] The multiple utterances 312 may include a first utterance 312A, a second utterance 312B, a third utterance 312C..., and utterance K 312K. An utterance (such as the first utterance 312A) can be defined as part of a dialogue or conversation that can be represented through a combination of speech sounds and images (or multiple images) and a transcript of the speech sounds. Each of the multiple utterances 312 may have a corresponding first modality related to the acoustics of the corresponding utterance, a second modality among the multiple modalities related to the transcript of the corresponding utterance, and a third modality related to the visual aspects of the corresponding utterance (e.g., facial expressions and lip movements). For example, a first utterance 312A may include a first modality 314A related to the acoustics of the first utterance 312A, a second modality 314B related to the transcription of the first utterance 312A, and a third modality 314C related to the visual aspects of the first utterance 312A. The first modality 314A, the second modality 314B, and the third modality 314C together can form a multimodal input (such as a multimodal input 124).

[0049] In one embodiment, a scene (or dialogue) may include k utterances “U” and their respective emotion labels “Y”, which can be arranged together in time. Each utterance may be accompanied by a corresponding first modality (i.e., speech segment), a second modality (i.e., transcript), and a third modality (i.e., video clip). As an example, a scene of k utterances can be mathematically represented using the following equation (1): TIFF0007864295000001.tif13150(1) Here, x i This represents the i-th utterance, TIFF0007864295000002.tif7150 represents the sound associated with the i-th utterance, TIFF0007864295000003.tif13150 represents the transcript related to the i-th utterance. TIFF0007864295000004.tif7150 represents a video related to the i-th utterance. y i This represents the emotion label of the i-th utterance.

[0050] The circuit 104 can be configured to input a multimodal input to one or more feature extractors 304. For example, the multimodal input may be related to a first utterance 312A depicted in one or more received videos.

[0051] According to one embodiment, the multimodal input may include a multilingual speech and a transcription of the multilingual speech in a first language that can be fitted to one or more feature extractors 304. For example, the multilingual speech may correspond to the utterance, "Hello Fred, Genkidesu ka?". In such a case, the transcription of the multilingual speech may be "Hello Fred, how are you?".

[0052] According to another embodiment, the multimodal input may include speech in a second language which may differ from the first language to which one or more feature extractors 304 are adapted. In such a case, the multimodal input may include a transcript of the speech in the first language (to which one or more feature extractors 304 are adapted). In such an embodiment, the circuit 104 may be configured to translate the speech from the second language to the first language by applying the translator model 208 to the speech (in the second language). The translation may be performed to overcome language compatibility issues between the speech (in the second language) and one or more feature extractors 304.

[0053] One or more feature extractors 304 may include an acoustic feature extractor 304B, a text feature extractor 304C, a visual feature extractor 304D, and an acoustic-visual feature extractor 304C. In one embodiment, each of the one or more feature extractors 304 may include at least one neural network that can be configured to extract features related to the corresponding modality. For example, the acoustic feature extractor 304B and the acoustic-visual feature extractor 304C may be configured to extract features related to the acoustics of the first utterance 312A. Similarly, the text feature extractor 304C may be configured to extract features related to the transcription of the first utterance 312A, and the visual feature extractor 304D and the acoustic-visual feature extractor 304C may be configured to extract features related to the visual aspects of the first utterance 312A. Such features can be collectively referred to as input embeddings.

[0054] In one embodiment, the circuit 104 can be configured to generate input embeddings based on applying one or more feature extractors 304 to a multimodal input. Specifically, the circuit 104 can be configured to generate a first embedding (FIA) based on applying an acoustic-visual feature extractor 304C to the acoustic information of utterances contained in the multimodal input. The acoustic-visual feature extractor 304C can be based on a triplet network that allows the acoustic-visual feature extractor 304C to leverage the importance of three loss functions. The acoustic-visual feature extractor 304C can include an encoder network and a projector module and can be trained on loss functions such as adaptive margin triplet loss, covariance loss, and variance loss. Details of the acoustic-visual feature extractor 304C are shown, for example, in Figure 6.

[0055] In one embodiment, circuit 104 may be configured to perform one or more operations on acoustic information before the acoustic information is provided as input to the acoustic-visual feature extractor 304C. Circuit 104 may be configured to convert the audio portion contained in the acoustic information into a two-dimensional (2D) Mel spectrogram in RGB format. Such a two-dimensional Mel spectrogram in RGB format can be provided as input to the acoustic-visual feature extractor 304C. Circuit 102 may be configured to process the audio signal of the audio portion through one or more augmentation techniques such as time warping and additive white Gaussian noise (AWGN) to generate augmented audio signals in order to convert the audio portion into a two-dimensional (2D) Mel spectrogram. Such generated augmented signals can be further converted into a Mel spectrogram. In one embodiment, a Mel spectrogram can be computed using a short-time Fourier transform (STFT) with a frame length of 400 samples (25 ms) and a hop length of 160 samples (10 ms), and 128 Mel filter banks. The circuit performs a first embedding (F) based on the application of the Mel spectrogram. IA ) can be further configured to generate

[0056] In another embodiment, the circuit 104 performs a first embedding (F) based on applying an acoustic feature extractor 304B to the acoustic information of the speech included in the multimodal input. IAThe acoustic feature extractor 304B can be configured to generate (F) input embeddings. The acoustic feature extractor 304B can be based on, for example, the openSMILE (open-source Speech and Music Interpretation by Large-space Extraction) model. In one embodiment, the acoustic feature extractor 304B may further include a multi-layer perceptron (MLP) network that can be trained on utterance labels. The circuit 104 can perform input embeddings (F) based on applying the multi-layer perceptron (MLP) network to the output generated by the acoustic feature extractor 304B using openSMILE. IA It can be configured to generate ). For example, the first embedding can be expressed mathematically using the following equation (2): TIFF0007864295000005.tif13150(2) Here, TIFF0007864295000006.tif13150 represents the first embedding, TIFF0007864295000007.tif7150 represents the operation of the acoustic video feature extractor 304C or the acoustic feature extractor 304B. k represents the total number of utterances. TIFF0007864295000008.tif7150 represents the sound (or audio component) of the i-th utterance. D A This represents the size of the embedded speech utterance.

[0057] In one embodiment, the circuit 104 can be configured to generate a second embedding of the input embedding based on applying the text feature extractor 304C to the transcription of acoustic information related to the first utterance 312A. The second embedding can further be generated based on applying the text feature extractor 304C to the transcription of different utterances that temporally precede or follow the first utterance 302A. The transcription of different utterances that temporally precede or follow the first utterance 302A is based on separator tokens ( <s>It can be separated by ). For example, if the fourth utterance in a scene is "The whole thing! Can we go?", the fifth utterance in the scene is "What about the scene with the kangaroo?", and the sixth utterance in the scene is "I was surprised to see a kangaroo in a world war epic", then the transcription of the fifth utterance is "The whole thing! <s>What about the scene with the Kangaroo <s>"I was surprised to see a kangaroo in a world war epic."

[0058] In one embodiment, the text feature extractor 304C can be implemented based on the RoBERTa model (Robustly optimized BERT (Bidirectional Encoder Representations from Transformers) method). For example, the second embedding can be mathematically expressed using the following equation (3): TIFF0007864295000009.tif13150(3) Here, TIFF0007864295000010.tif13150 represents the second embedding of the input embedding. TIFF0007864295000011.tif7150 represents the operation of the text feature extractor 304C (RoBERTa model or modified RoBERTa model), k represents the total number of utterances. TIFF0007864295000012.tif13150 represents the transcript of the i-th utterance. D T This represents the embedding size of the text utterance.

[0059] In one embodiment, the circuit 104 can be configured to generate a third embedding of the input embedding based on applying either the acoustic-visual feature extractor 304C or the visual feature extractor 304D to the face information of one or more speaking characters in one or more video frames, and scene information associated with the frame. The frame is the first utterance 312A in one or more videos Duration This can be addressed. Each of the acoustic-visual feature extractor 304C and the visual feature extractor 304D can be a dual network that can be configured to detect, for example, one or more faces of one or more utterance characters and the area of ​​each of the one or more faces. In one embodiment, the acoustic-visual feature extractor 304C or the visual feature extractor 304D may include a first network that extracts features from one or more detected features of one or more utterance characters and a second network that extracts features from the entire scene containing one or more utterance characters. The visual feature network can normalize the detected one or more faces based on their corresponding areas to generate a third embedding of the input embedding. Details of the acoustic-visual feature extractor 304C and the visual feature extractor 304D are shown, for example, in Figure 4. The generated third embedding can be expressed mathematically using the following equation (4): TIFF0007864295000013.tif13150(4) Here, TIFF0007864295000014.tif13150 represents the third embedding of the input embedding, TIFF0007864295000015.tif7150 represents the operation of the acoustic-visual feature extractor 304C or the visual feature extractor 304D. k represents the total number of utterances. TIFF0007864295000016.tif7150 represents a video related to the i-th utterance. D V This represents the embedding size of the visual utterance.

[0060] The generated input embeddings (including embeddings for each modality of the multimodal input) can be supplied as input to the transformer encoder network 306. The input embeddings can be passed through the transformer encoder network 306 to learn the inter-utterance context for each modality of the multimodal input. The transformer encoder network 306 may include a first stack of transformer encoders 316 for the first modality 314A of the multimodal input, a second stack of transformer encoders 318 for the second modality 314B of the multimodal input, and a third stack of transformer encoders 320 for the third modality 314C of the multimodal input. In some embodiments, the first stack of transformer encoders 316, the second stack of transformer encoders 318, and the third stack of transformer encoders 320 may each contain the same number of transformer encoders. In another embodiment, the first stack 316 of transformer encoders, the second stack 318 of transformer encoders, and the third stack 320 of transformer encoders may include a different number of transformer encoders.

[0061] The first stack 316 of the transformer encoder can include from the first transformer encoder 316A... to the Nth transformer encoder 316N. The output of the first transformer encoder 316A can be provided as an input to the Nth transformer encoder 316N. Similarly, the second stack 318 of the transformer encoder can include from the first transformer encoder 318A... to the Nth transformer encoder 318N. The output of the first transformer encoder 318A can be provided as an input to the Nth transformer encoder 318N. Similarly, the third stack 320 of the transformer encoder can include from the first transformer encoder 320A... to the Nth transformer encoder 320N. The output of the first transformer encoder 320A can be provided as an input to the Nth transformer encoder 320N.

[0062] The first embedding (F IA ) of the input embedding can be provided as an input to the first transformer encoder 316A of the first stack 316 of the transformer encoder. The second embedding (F IT ) of the input embedding can be provided as an input to the first transformer encoder 318A of the second stack 318 of the transformer encoder. Similarly, the third embedding (F IV ) of the input embedding can be provided as an input to the first transformer encoder 318A of the third stack 320 of the transformer encoder.

[0063] Each transformer encoder of the transformer encoder network 306 can be trained to generate an emotion-related feature set. The emotion-related feature set can include one or more features corresponding to each modality of the multimodal input. For example, one or more features corresponding to the first modality 314A can be mathematically represented using the following equation (5), TIFF0007864295000017.tif13150(5) Here, TIFF0007864295000018.tif13150 represents the first embedding of the input embedding, T δ This represents the operation of the transformer encoder. N1 represents the first transformer encoder 316A of the first stack 316 of the transformer encoder, N A represents the Nth transformer encoder 316N of the first stack 316 of the transformer encoder, i ∈ [1, k].

[0064] One or more features corresponding to the second modality 314B can be mathematically expressed using the following equation (6): TIFF0007864295000019.tif13150(6) Here, TIFF0007864295000020.tif13150 represents the second embedding of the input embedding, T δ This represents the operation of the transformer encoder. N1 represents the first transformer encoder 318A of the second stack 318 of the transformer encoder, N T represents the Nth transformer encoder 318N of the second stack 318 of the transformer encoder, i ∈ [1, k]. Similarly, one or more features corresponding to the third modality 314C can be mathematically expressed using the following equation (7): TIFF0007864295000021.tif13150(7) Here, TIFF0007864295000022.tif13150 represents the third embedding of the input embedding, T δ This represents the operation of the transformer encoder. N1 represents the first transformer encoder 320A of the third stack 320 of the transformer encoder, N V This represents the Nth transformer encoder 318N of the third stack 320 of the transformer encoder, i ∈ [1, k].

[0065] In one embodiment, the multimodal fusion network 302 may further include skip connections 322 between each pair of adjacent transformer encoders in the transformer encoder network 306. Specifically, the skip connections 322 may exist between each pair of adjacent transformer encoders in the first stack 316 of transformer encoders, the second stack 318 of transformer encoders, and the third stack 320 of transformer encoders. The skip connections 322 may be employed within the multimodal fusion network 302 to prevent the multimodal fusion network 302 from ignoring lower-level features associated with each modality of the multimodal input.

[0066] The generated emotion-related feature set can be provided as input to the fusion attention network 308 of the fusion attention network set 308. The fusion attention network set 308 can be coupled to the transformer encoder network 306 and may include at least one fusion attention network. For example, the fusion attention network set 308 may include, but is not limited to, first fusion attention networks 308A... and up to Nth fusion attention networks 308N. Each fusion attention network in the fusion attention network set 308 may include one or more multi-head attention layers and a first fully connected layer. In one embodiment, the input of the first fully connected layer may be coupled to the output of one or more multi-head attention layers of the corresponding fusion attention network. Each of the fusion attention network sets 308 may be configured to output a fused feature representation of the emotion-related feature set. Details regarding each of the fusion attention network sets 308 and the emotion-related feature set are shown, for example, in Figure 5. As an example, not an exhaustive one, the fused feature representation of a set of emotion-related features can be mathematically expressed using the following equations (8) and (9): TIFF0007864295000023.tif13150(8) TIFF0007864295000024.tif13150(9) Here, TIFF0007864295000025.tif13150 represents the output of the first Fusion Attention Network 308A, MHA represents the operation of one or more multi-head attention layers. TIFF0007864295000026.tif13150 represents the output of the Nth Fusion Attention Network 308N (or the output of Fusion Attention Network Set 308), TIFF0007864295000027.tif13150 represents one or more features corresponding to the first modality 314A, TIFF0007864295000028.tif13150 represents one or more features corresponding to the second modality 314B, TIFF0007864295000029.tif13150 represents one or more features corresponding to the third modality 314C, m represents the total number of one or more multi-head attention layers in the Fusion Attention Network Set 308. i ∈ [1, k].

[0067] In one embodiment, the generated fused feature representation can be provided as input to the output network 310. The output network 310 may include a second fully connected layer that can be coupled to the output of the fusion attention network set 308. The second fully connected layer of the output network 310 may be configured to predict the emotion label of a first utterance 312A among a plurality of utterances 312. In one embodiment, the second fully connected layer may include a SoftMax function or a cross-entropy function implemented at the output of the second fully connected layer. The predicted emotion label may be, but is not limited to, one of the following: happy emotion label, sad emotion label, angry emotion label, calm emotion label, frightened emotion label, neutral emotion label, excited emotion label, confused emotion label, stressed emotion label, nauseated emotion label, surprised emotion label, excited emotion label, or frightened emotion label.

[0068] In one embodiment, the output network 310 can be configured to predict the sentiment label of a multimodal input. The predicted sentiment label can indicate whether the utterance (for which the multimodal input is provided as input to the multimodal fusion network 302) corresponds to a positive, negative, or neutral sentiment.

[0069] In one embodiment, the output of the output network 310 can be mathematically expressed using the following equations (10) and (11): TIFF0007864295000030.tif13150(10) TIFF0007864295000031.tif13150(11) Here, TIFF0007864295000032.tif13150 represents the predicted sentiment label for each of the 312 utterances. TIFF0007864295000033.tif13150 represents the predicted sentiment label for the first utterance 312A among multiple utterances 312. TIFF0007864295000034.tif13150 represents the predicted sentiment label for the p-th utterance out of 312 utterances. FC represents the operation of the second fully connected layer of the output network 310. TIFF0007864295000035.tif13150 represents the output of the Nth Fusion Attention Network 308N (or the output of Fusion Attention Network Set 308), i ∈ [1, k].

[0070] In one embodiment, the operation of the multimodal fusion network 302 can be divided into two levels: the utterance level and the dialogue level. Since embeddings associated with each modality of the multimodal input can be generated independently, one or more feature extractors 304 can be considered part of the utterance level. At the dialogue level, the multimodal fusion network 302 can learn to predict the sentiment of each utterance by using contextual information from the entire dialogue.

[0071] In one embodiment, the multiple modalities may include a fourth modality related to one or more biological parameters of the subject associated with the corresponding utterance. For example, one or more biological parameters can be captured during utterance recording and converted into time-series data (or multidimensional spatial data) for further processing by the multimodal fusion network 302. Circuit 104 can be configured to predict the subject's emotion label based on all four modalities, i.e., the first modality, the second modality, the third modality, and the fourth modality. Below is pseudocode of the operation performed by system 102: TIFF0007864295000036.tif149150 Here, MHA supports multi-head attention network operation (i.e., fusion attention network operation), Concat supports concatenation operations. FC corresponds to the operation of the second fully coupled layer, D T This corresponds to the dimensions of the text.

[0072] Figure 4 shows an exemplary visual feature extractor of the multimodal fusion attention network of Figure 3 according to an embodiment of the present disclosure. The description of Figure 4 will be made in relation to the elements of Figures 1, 2, and 3. Figure 4 contains a system 400 which may include a system 402. System 402 may be an exemplary implementation of system 102 of Figure 1. System 402 includes a multimodal fusion network 302. The multimodal fusion network 302 may include one or more feature extractors 304. Figure 4 shows a first utterance 312A of a plurality of utterances 312 of a plurality of frames. Duration The corresponding frame 404 is shown further.

[0073] System 402 can input multiple frames from one or more videos into the visual feature extractor 304D. The visual feature extractor 304D may include a face detection model 210. In one embodiment, the face detection model 210 may correspond to a Multi-task Cascaded Convolutional Network (MTCNN). The face detection model 210 can be applied to frame 414 to detect one or more faces in each of the received multiple frames. For example, in frame 404, the detected one or more faces may include a first face 406 and a second face 408.

[0074] The system 402 can be configured to generate one or more bounding boxes that can include one or more detected faces. The generated one or more bounding boxes may include a first bounding box 410 and a second bounding box 412. The first bounding box 410 may include a first face 406, and the second bounding box 412 may include a second face 408. In one embodiment, one or more bounding boxes can be generated based on applying the face detection model 210 to the frame 404. The system 402 can then normalize the areas associated with each of the one or more bounding boxes based on the application of either the acoustic-visual feature extractor 304C or the visual feature extractor 304D. Based on the normalization, the system 402 can generate a third embedding of the input embedding as the output of the acoustic-visual feature extractor 304C or the visual feature extractor 304D.

[0075] System 402 can be configured to determine a weighted sum based on the features associated with each of the one or more detected faces and the corresponding normalized area. The weighted sum can be expressed mathematically using the following equation (12): TIFF0007864295000037.tif7150(12) Here, F IV This represents the third embedding of the input embeddings, F1 represents the features associated with the first detected face 406, F2 represents features associated with the second detected face 408, W1 represents the normalized area of ​​the first bounding box 410, W2 represents the normalized area of ​​the second bounding box 412.

[0076] In one embodiment, the visual feature extractor 304D extracts the first utterance 312A Duration The system may include a visual transformer (ViT) that can be used for a first total number of consecutive frames spanning (e.g., 15 in the case of acoustic-visual feature extractor 304C, and 30 in the case of visual feature extractor 304D). Based on the operations described above, the system 102 can extract features from each frame included in the first total number. The features extracted from each of the consecutive frames in the first total number can be pooled to generate a third embedding (expressed using equations (7) and (12)).

[0077] In another embodiment, the acoustic-visual feature extractor 304C or the visual feature extractor 304D may be a dual network. The dual network may include a first network that detects one or more faces in frame 404 and a second network that focuses on the entire frame 404. Specifically, the second network may focus on one or more objects and other visual cues (i.e., scene information) that are visible in frame 404. The circuit 104 may be further configured to generate a third embedding of the input embedding as the output of the acoustic-visual feature extractor 304C or the visual feature extractor 304D, based on applying the first and second networks to (single and multiple) frames.

[0078] Figure 5 shows an exemplary architecture of the fusion attention network of Figure 3 according to an embodiment of the present disclosure. The description of Figure 5 will be made in relation to the elements of Figures 1, 2, 3 and 4. Figure 5 shows Figure 500 of a first attention network 502 which may be an exemplary embodiment of the first fusion attention network 308A of the fusion attention network set 308.

[0079] The first attention network 502 may include one or more multi-head attention layers, which may further include a first multi-head attention layer 504A and a second multi-head attention layer 504B. The first attention network 502 may further include a fully connected layer 506 (also called the first fully connected layer). The architecture of each fusion attention network in the fusion attention network set 308 may be the same as that shown in Figure 500. The fusion attention network set 308 may be coupled to a transformer encoder network 306 and an output network 310. Specifically, the output of the transformer encoder network 306 (i.e., a set of emotion-related features) may be provided as input to the fusion attention network set 308 (specifically, the first fusion attention network 308A), and the output of the fusion attention network set 308 may be provided as input to the output network 310.

[0080] Circuit 104 can be configured to provide an emotion-related feature set as input to the first attention network 502 of the fusion attention network set 308. As described above, the emotion-related feature set may include one or more features related to the first modality (represented by equation (5)), one or more features related to the second modality (represented by equation (6)), and one or more features related to the third modality (represented by equation (7)). Specifically, circuit 104 can be configured to provide an emotion-related feature set to one or more multi-head attention layers of the first attention network 502.

[0081] Each of the one or more multi-head attention layers can accept queries, keys, and values ​​as input and can be configured to capture dependencies of various ranges (e.g., short ranges and long ranges) within a sequence. In one embodiment, one or more features (F) related to the first modality A ) is designated as the "key (k)", and one or more features (F) related to the second modality are used. T ) can be provided to the first multi-head attention layer 504A as a "query (q) and value (v)". Similarly, one or more features (F) related to the third modality can be provided. V ) is used as the "key (k)", and one or more features (F) related to the third modality. T ) can be provided to the second multi-head attention layer 504B as a "query (q) and value (v)".

[0082] Circuit 104 can be configured to apply one or more multi-head attention layers to the sentiment-related feature set to determine the feature mappings within the sentiment-related feature set. As described, each of the one or more multi-head attention layers can capture the dependencies between (single / multiple) features associated with a first modality, (single / multiple) features associated with a second modality, and (single / multiple) features associated with a third modality. Mapping can be used to map each of the multiple modalities into the text vector space. Circuit 104 can be configured to concatenate the sentiment-related feature set into a latent representation of the sentiment-related feature set based on the feature mappings. After concatenation, the concatenated output (i.e., the latent representation of the sentiment-related feature set) can be provided as input to the fully connected layer 506. Circuit 104 can be configured to generate a fused feature representation of the sentiment-related feature set based on the application of the fully connected layer 506. The fused feature representation of the sentiment-related feature set is R k*D T It can be a vector that belongs to and can be expressed using equations (8) and (9). Specifically, the vector has dimension K * D T It can belong to the real coordinate space.

[0083] In one embodiment, circuit 104 can be further configured to provide a fused feature representation of a set of sentiment-related features as input to a second fully connected layer of output network 310, which can be coupled to the output of fusion attention network set 308. Circuit 104 can be further configured to predict the sentiment label of a corresponding utterance based on the application of the second fully connected layer of output network 310. In one embodiment, the output of the first fusion attention network 308A can be passed to the second fusion attention network. This same process can be repeated n times to pass the output of the nth fusion attention network 308N as input to a second fully connected layer (i.e., output network 310) which can be configured to predict the sentiment label of a corresponding utterance.

[0084] Figure 6 shows an exemplary architecture of an acoustic-visual feature extractor of one or more feature extractors 304 of Figure 3 according to an embodiment of the present disclosure. The description of Figure 6 will be made in relation to the elements of Figures 1, 2, 3, 4, and 5. Figure 6 shows Figure 600 of an acoustic-visual feature extractor 602 of one or more feature extractors 304.

[0085] The acoustic-visual feature extractor 602 may be based on a triplet network, which may require three input samples. Further shown is an encoder network set 604 which may include a first encoder network 604A, a second encoder network 604B, and a third encoder network 604C. The acoustic-visual feature extractor 602 may further include a projector set 606, which may include a first projector 606A, a second projector 606B, and a third projector 606C.

[0086] In one embodiment, input samples 608 related to acoustic and visual modalities can be divided into a positive sample set 610A, an anchor sample set 610B, and a negative sample set 610C. Each of the positive sample sets 610A may be the same as the anchor sample set 610B, and each of the positive sample sets 610A may be different from the anchor sample set 610B. The positive sample set 610A can be supplied to a first encoder network 604A. The anchor sample set 610B can be supplied to a second encoder network 604B. Similarly, the negative sample set 610C can be supplied to a third encoder network 604C. Each encoder in the encoder set 604 may have the same architecture, with the same number of neurons and associated weights. An example of an encoder may be a ResNet-18 network.

[0087] In one embodiment, a first encoder network 604A can generate a first output. A second encoder network 604B can generate a second output. Similarly, a third encoder network 604C can generate a third output. The first output can be provided as input to a first projector 606A, the second output can be provided as input to a second projector 606B, and the third output can be provided as input to a third projector 606C. Each projector in the projector set 606 may include a fully linear-fully connected layer that can be configured to project the embedding of the encoder network set 604 onto the representation set 612. Specifically, the representation set 612 may include a first representation 612A of the positive sample set 610A, a second representation 612B of the anchor sample set 610B, and a third representation 612C of the negative sample set 610C. The representation set 612 can be mathematically represented using the following equation (13): TIFF0007864295000038.tif13150(13) Here, Z represents the desired expression, N represents the total number of representations. 'd' represents the dimension of each representation.

[0088] The acoustic-visual feature extractor 602 can be trained using a weighted combination of three loss functions: an adaptive margin triplet loss function, a covariance loss function, and a variance loss function. The objective of training the acoustic-visual feature extractor 602 can be to decrease the distance between the positive sample set 610A and the anchor sample set 610B, and to increase the second distance between the anchor sample set 610B and the negative sample set 610C. The weighted combination of the three loss functions can be expressed mathematically using the following equation (14): TIFF0007864295000039.tif7150(14) Here, λ1, λ2, and λ3 represent weighting coefficients, L AMT This represents the adaptive margin triplet loss function, L Cov This represents the covariance loss function, L Var This represents the variance loss function, L FE This represents the triplet loss function.

[0089] Traditionally, developers have designed triplet loss functions used to learn good facial representations based on a positive sample set 610A, an anchor sample set 610B, and a negative sample set 610C. Developers tend to use fixed margin values ​​in their triplet loss functions to help separate positive and negative sample representations. However, when the distance from a positive or negative sample to an anchor is the same, or when a positive sample is slightly closer to an anchor than a negative sample, the triplet loss calculated for such a fixed margin can be zero, resulting in no correction despite the expectation that positive samples should be closer to the anchor and negative samples further away. To address this issue, an adaptive margin value loss function can be used in the calculation of the triplet loss function. This adaptive margin value loss function can be mathematically expressed using the following equation (15): TIFF0007864295000040.tif13150(15) Here, TIFF0007864295000041.tif7150 represents a Euclidean distance-based similarity metric between the representation in positive sample set 610A and the representation in anchor sample set 610B. TIFF0007864295000042.tif7150 represents a Euclidean distance-based similarity metric between the representation in anchor sample set 610B and the representation in negative sample set 610C. TIFF0007864295000043.tif7150 represents a Euclidean distance-based similarity metric between the representations in the positive sample set 610A and the negative sample set 610C. m AM This represents the adaptive margin.

[0090] In one embodiment, the adaptive margin (m AM ) can be calculated based on similarity and dissimilarity scales, and can be expressed mathematically using the following equation (16): TIFF0007864295000044.tif13150(16) Here, TIFF0007864295000045.tif7150 represents a similarity scale, The filename is TIFF0007864295000046.tif13150. TIFF0007864295000047.tif13150 represents a similarity scale, The filename is TIFF0007864295000048.tif13150.

[0091] In one embodiment, the triple loss function may also include a variance loss function. The variance loss function can help the acoustic-visual feature extractor 602 deal with the (single and multiple) mode collapse issue, and can be mathematically expressed using the following equation (17): TIFF0007864295000049.tif39150(17) Here, Var(Z) represents the variance obtained from the corresponding representation, The filename is TIFF0007864295000050.tif20150. Z k = Z A ,Z p ,Z n And, Z p This represents the first representation corresponding to the positive sample set 610A, Z a This represents the second representation corresponding to anchor sample set 610B, Z n This represents the third representation corresponding to the negative sample set 610C, TIFF0007864295000051.tif7150 represents the average of the corresponding expressions.

[0092] In one embodiment, the triple loss function may also include a covariance loss function. The covariance loss function can help the acoustic-visual feature extractor 602 decorrelate different dimensions of the representation and can be mathematically expressed using the following equation (18): TIFF0007864295000052.tif39150(18) Here, Cov(Z) represents the covariance matrix of the corresponding representation, The filename is TIFF0007864295000053.tif20150. Z k = Z A ,Z p ,Z n And, Z p This represents the first representation corresponding to the positive sample set 610A, Z a This represents the second representation corresponding to anchor sample set 610B, Z n This represents the third representation corresponding to the negative sample set 610C, TIFF0007864295000054.tif7150 represents the average of the corresponding expressions.

[0093] Figure 7 shows an exemplary scenario for emotion recognition in multimedia video using a multimodal fusion-based deep neural network according to embodiments of the present disclosure. The description of Figure 7 will be made in relation to the elements of Figures 1, 2, 3, 4, 5, and 6. Figure 7 shows scenario 700. Scenario 700 shows a system 102 including the circuit 104 of Figure 1 and the multimodal fusion network 108 of Figure 3. Furthermore, it shows a plurality of multimodal inputs 702 and a plurality of predicted emotion labels 704.

[0094] Multiple multimodal inputs 702 may include a first multimodal input 702A, a second multimodal input 702B, a third multimodal input 702C, and an Nth multimodal input 702N. The first multimodal input 702A may relate to a first utterance depicted in one or more videos. The second multimodal input 702B may relate to a second utterance depicted in such a video. Similarly, the third multimodal input 702C may relate to a third utterance depicted in such a video, and the Nth multimodal input 702N may relate to an Nth utterance depicted in such a video. All such utterances may be part of a conversation (e.g., a two-person conversation). Similar to the multiple multimodal inputs 702, the multiple predicted sentiment labels 704 may include a first predicted sentiment label 704A, a second predicted sentiment label 704B, a third predicted sentiment label 704C, and N predicted sentiment labels 704N.

[0095] The circuit 104 can be configured to input the first multimodal input 702A to one or more feature extractors 110. The first multimodal input 702A may be related to a first utterance and may include a first modality 706 related to the acoustics of the first utterance, a second modality 708 related to the transcription of the first utterance, and a third modality 710 related to the visual aspects of the first utterance.

[0096] The circuit 104 can be further configured to generate input embeddings as the output of one or more feature extractors 110 for the input. The input embeddings include embeddings for each modality of the multimodal input. Details regarding the input embeddings are shown, for example, in Figure 3.

[0097] Circuit 104 can be further configured to generate a set of sentiment-related features based on applying a transformer encoder network to the input embedding. The set of sentiment-related features may include one or more features corresponding to each modality of the multimodal input. After generating the set of sentiment-related features, Circuit 104 can be configured to generate a fused feature representation of the set of sentiment-related features. In one embodiment, the fused feature representation of the set of sentiment-related features can be generated based on applying a fusion attention network to the set of sentiment-related features. Circuit 104 can be further configured to output a first predicted sentiment label 704A of the first utterance based on applying an output network 116 to the fused feature representation.

[0098] In one embodiment, the above-described operation can be performed for each of a plurality of multimodal inputs 702 to predict the corresponding sentiment label for the corresponding multimodal input. For example, the first predicted sentiment label 704A for the first utterance may be "surprise." The sentiment associated with the first predicted sentiment label 704A may be "positive." The second predicted sentiment label 704B for the second utterance may be "joy," and the corresponding sentiment may be "positive." The third predicted sentiment label 704C for the third utterance may be "neutral," and the corresponding sentiment may be "neutral." Similarly, the Nth predicted sentiment label 704N for the Nth utterance may also be "neutral," and the corresponding sentiment may also be "neutral."

[0099] Figure 8 is a flowchart illustrating an exemplary emotion recognition method in multimedia video using a multimodal fusion-based deep neural network according to an embodiment of the present disclosure. The description of Figure 8 will be made in relation to the elements of Figures 1, 2, 3, 4, 5, 6, and 7. Figure 8 shows flowchart 800. The operation of flowchart 800 can be started at 802 and proceeded to 804.

[0100] In 804, one or more feature extractors 110 can receive multimodal inputs that may be related to utterances depicted in one or more videos. In at least one embodiment, the circuit 104 can be configured to receive one or more feature extractors that may receive multimodal inputs that may be related to utterances depicted in one or more videos. Details regarding the multimodal inputs are shown, for example, in Figures 1, 3, and 5.

[0101] In 806, input embeddings can be generated that include embeddings for each modality of the multimodal input as the output of one or more feature extractors 110 to the input. In at least one embodiment, the circuit 104 can be configured to generate input embeddings that include embeddings for each modality of the multimodal input as the output of one or more feature extractors to the input. Details regarding the generation of input embeddings are shown, for example, in Figure 3.

[0102] In 808, a set of sentiment-related features can be generated that includes one or more features corresponding to each modality of the multimodal input, based on the application of a transformer encoder network to the input embedding. In at least one embodiment, the circuit 104 can be configured to generate a set of sentiment-related features that includes one or more features corresponding to each modality of the multimodal input, based on the application of a transformer encoder network 112 to the input embedding. Details regarding the generation of the sentiment-related feature set are shown, for example, in Figure 3.

[0103] In 810, a fused feature representation of the emotion-related feature set can be generated based on applying a fusion attention network to the emotion-related feature set. In at least one embodiment, the circuit 104 can be configured to generate a fused feature representation of the emotion-related feature set based on applying a fusion attention network to the emotion-related feature set. Details regarding the generation of the fused feature representation are shown, for example, in Figures 3 and 5.

[0104] In 812, the sentiment label of an utterance can be predicted based on applying the output network 116 to the fused feature representation. In at least one embodiment, the circuit 104 can be configured to predict the sentiment label of an utterance based on applying the output network to the fused feature representation. Control can then proceed to termination.

[0105] Experimental data obtained after conducting multiple experiments showed that the disclosed multimodal fusion network 302 significantly outperformed state-of-the-art methods (i.e., improved by more than 9% in terms of weighted average F1 score) when run on known datasets such as the Multimodal Multi-Party Dataset for Emotion Recognition in Conversation (MELD) dataset and the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset.

[0106] Various embodiments of this disclosure can provide a non-temporary, computer-readable medium storing computer-executable instructions that a circuit or machine can execute to operate a system (e.g., system 102) for emotion recognition in multimedia video using a multimodal fusion-based deep neural network. The computer-executable instructions can cause a machine and / or computer to perform an operation which includes inputting a multimodal input (e.g., multimodal input 124) to one or more feature extractors (e.g., one or more feature extractors 110) of a multimodal fusion network (e.g., multimodal fusion network 108). The multimodal input may relate to utterances depicted in one or more videos. The operation may further include generating input embeddings as outputs of one or more feature extractors to the input. The input embeddings may include embeddings for each modality of the multimodal input. The operation may further include generating emotion-related feature sets based on applying a transformer encoder network (e.g., transformer encoder network 112) of the multimodal fusion network to the input embeddings. The emotion-related feature set includes one or more features corresponding to each modality of the multimodal input. The operation may further include generating a fused feature representation of the emotion-related feature set based on applying the fusion attention network of the multimodal fusion network (e.g., fusion attention network 114) to the emotion-related feature set. The operation may further include predicting the emotion label of the utterance based on applying the output network of the multimodal fusion network (e.g., output network 116) to the fused feature representation.

[0107] Several embodiments of the present disclosure can be found in systems and methods for emotion recognition in multimedia video using multimodal fusion-based deep neural networks. Various embodiments of the present disclosure can provide a system 102 which may include a circuit 104 and a memory 106, the memory 106 being configured to store a multimodal fusion network 108 which includes one or more feature extractors 110, a transformer encoder network 112 coupled to one or more feature extractors 110, a fusion attention network 114 coupled to the transformer encoder network 112, and an output network 116 coupled to the fusion attention network 114. The circuit 104 may be configured to input a multimodal input 124 to one or more feature extractors. The multimodal input may relate to utterances depicted in one or more videos. The circuit 104 may be further configured to generate input embeddings as outputs of one or more feature extractors 110 to the input. The input embeddings may include embeddings for each modality of the multimodal input. Circuit 104 can be further configured to generate a set of sentiment-related features based on applying a transformer encoder network 112 to the input embedding. The sentiment-related feature set includes one or more features corresponding to each modality of the multimodal input. Circuit 104 can be further configured to generate a fused feature representation of the sentiment-related feature set based on applying a fusion attention network 114 to the sentiment-related feature set. Circuit 104 can be further configured to predict the sentiment label of an utterance based on applying an output network 310 to the fused feature representation.

[0108] According to one embodiment, the multimodal input 124 includes a multilingual speech and a transcript of the multilingual speech in a first language that can be adapted to one or more feature extractors. According to one embodiment, the multimodal input includes speech in a second language that may differ from the first language adapted to one or more feature extractors 110, and the multimodal input includes a transcript of the speech in the first language that can be adapted to one or more feature extractors 110. According to one embodiment, the multimodal input includes a first modality 314A related to the acoustics of the speech, a second modality 314B related to the transcription of the speech, and a third modality 314C related to the visual aspects of the speech.

[0109] According to one embodiment, one or more feature extractors may include an acoustic feature extractor 304B and an acoustic-visual feature extractor 304C, and the circuit 104 may be further configured to generate a first embedding of the input embedding based on applying either the acoustic-visual feature extractor 304C or the acoustic feature extractor 304B to the acoustic information of the utterance contained in the multimodal input 124.

[0110] According to one embodiment, one or more feature extractors may include a text feature extractor 304C, and the circuit 104 may be further configured to generate a second embedding of the input embedding based on applying the text feature extractor 304C to the transcription of acoustic information related to the utterance and the transcription of different utterances that temporally precede or follow the utterance.

[0111] According to one embodiment, one or more feature extractors may include a visual feature extractor 304D and an acoustic-visual feature extractor 304C, and the circuit 104 may be further configured to generate a third embedding of the input embedding based on applying either the acoustic-visual feature extractor 304C or the visual feature extractor 304D to the face information of one or more speaking characters in the frames of one or more videos, and scene information associated with the frames. The frames are the utterances in one or more videos. Duration It can handle this.

[0112] According to one embodiment, the circuit 104 is used for speech Duration The circuit 104 can be configured to input one or more video frames corresponding to the input embedding to the visual feature extractor 304D. The circuit 104 can be further configured to detect one or more faces in each of the received frames by applying the face detection model 210 of the visual feature extractor 304D to each of the received frames. The circuit 104 can be further configured to generate one or more bounding boxes containing the detected one or more faces. The circuit 104 can be further configured to normalize the area associated with each of the one or more bounding boxes by applying the visual feature extractor 304D. The circuit 104 can be further configured to generate a third embedding of the input embedding as the output of the visual feature extractor 304D based on the detected one or more faces and the normalization.

[0113] According to one embodiment, the transformer encoder network 306 may include a first stack 316 of transformer encoders for a first modality 314A of a multimodal input, a second stack 318 of transformer encoders for a second modality 314B of a multimodal input, and a third stack 320 of transformer encoders for a third modality 314C of a multimodal input.

[0114] According to one embodiment, the system 102 may further include skip connections 322 between each pair of adjacent transformer encoders in the transformer encoder network 306.

[0115] According to one embodiment, circuit 104 can be configured to receive one or more videos. Circuit 104 can be further configured to apply a scene detection model 212 to the received one or more videos. Circuit 104 can be further configured to extract multiple scenes from one or more videos based on the application of the scene detection model. Circuit 104 can be further configured to apply a single boundary detection model 214 to each of the extracted multiple scenes. Circuit 104 can be further configured to detect multiple utterances 312 in the extracted multiple scenes based on the application of the single boundary detection model 214. Circuit 104 can be further configured to prepare a set of multimodal inputs based on the detections. The multimodal inputs that can be input to one or more feature extractors 304 can be part of the prepared set of multimodal inputs.

[0116] According to one embodiment, each of the fusion attention network sets 308 may include one or more multi-head attention layers and a first fully coupled layer. The inputs of the first fully coupled layer may be coupled to the outputs of one or more multi-head attention layers.

[0117] According to one embodiment, the circuit 104 can be further configured to apply one or more multi-head attention layers to the emotion-related feature set to determine the feature mappings within the emotion-related feature set, and to link the emotion-related feature set to a latent representation of the emotion-related feature set based on the feature mappings.

[0118] According to one embodiment, a fused feature representation of the emotion-related feature set can be further generated based on applying a first fully connected layer to the latent representation.

[0119] According to one embodiment, the output network 310 may include a second fully coupled layer coupled to the output of the fusion attention network set 308.

[0120] This disclosure can be implemented in hardware or in a combination of hardware and software. This disclosure can be implemented centrally within at least one computer system or in a distributed manner, where different elements can be distributed across multiple interconnected computer systems. A computer system or other device adapted to perform the methods described herein may be suitable. The hardware-software combination may be a general-purpose computer system including a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. This disclosure can also be implemented in hardware, including a portion of an integrated circuit that also performs other functions.

[0121] This disclosure includes all features that enable the implementation of the methods described herein and can be incorporated into a computer program product that can perform these methods when loaded onto a computer system. In this context, a computer program means any expression in any language, code, or notation of an instruction set intended to be executed directly, or after either a) conversion to another language, code, or notation, or b) reproduction in a different content form, on a system having information processing capabilities.

[0122] While this disclosure has been described with reference to several embodiments, those skilled in the art will understand that various modifications can be made and equivalents can be substituted without departing from the scope of this disclosure. Furthermore, many modifications can be made without departing from the scope of this disclosure to suit specific circumstances or content to the teachings of this disclosure. Accordingly, this disclosure is not limited to the specific embodiments disclosed, but is intended to include all embodiments that fall within the scope of the appended claims. [Explanation of symbols]

[0123] 210 Face Detection Models 302 Multimodal Fusion Network 304A Acoustic-Visual Feature Extractor 304D Visual Feature Extractor 402 System 404 frames 406 The First Face 408 The Second Face 408 First bounding box 412 Second bounding box< / s> < / s> < / s>

Claims

1. It is a system, A memory configured to store a multimodal fusion network including one or more feature extractors, including an acoustic-visual feature extractor and a visual feature extractor; a transformer encoder network coupled to the one or more feature extractors; a fusion attention network coupled to the transformer encoder network; and an output network coupled to the fusion attention network. Multimodal inputs related to speech depicted in one or more videos are input to the one or more feature extractors. As the output of the one or more feature extractors for the input, an input embedding is generated which includes an embedding for each modality of the multimodal input. Based on applying the transformer encoder network to the input embedding, a set of emotion-related features is generated that includes one or more features corresponding to each modality of the multimodal input. Based on applying the fusion attention network to the emotion-related feature set, a fused feature representation of the emotion-related feature set is generated. Based on applying the output network to the fused feature representation, predict the emotion label of the utterance. A circuit configured as follows, Equipped with, The circuit is further configured to generate a third embedding of the input embedding based on applying either the acoustic-visual feature extractor or the visual feature extractor to the facial information of one or more speaking characters within the frames of one or more videos, and to the scene information associated with the frames. The frame is a frame corresponding to the duration of the utterance in the one or more videos, The aforementioned circuit is The one or more video frames corresponding to the duration of the utterance are input to the acoustic-visual feature extractor or the visual feature extractor. Based on applying an acoustic-visual feature extractor or a face detection model of the visual feature extractor to each of the received frames, one or more faces are detected in each of the received frames. One or more bounding boxes are generated that include the one or more faces detected. By applying either the acoustic-visual feature extractor or the visual feature extractor, the areas associated with each of the one or more bounding boxes are normalized. Based on the detected one or more faces and the normalization, a third embedding of the input embedding is generated as the output of the acoustic-visual feature extractor or the visual feature extractor. A system characterized by being further configured in such a way.

2. The multimodal input includes multilingual speech and a transcription of the multilingual speech in a first language that is compatible with the one or more feature extractors. The system according to claim 1.

3. The multimodal input includes speech in a second language different from a first language that fits the one or more feature extractors, and the multimodal input includes a transcript of the speech in the first language that fits the one or more feature extractors. The system according to claim 1.

4. The multimodal input includes a first modality related to the acoustics of the utterance, a second modality related to the transcription of the utterance, and a third modality related to the visual aspects of the utterance. The system according to claim 1.

5. The one or more feature extractors include an acoustic-visual feature extractor and an acoustic feature extractor, and the circuit is further configured to generate a first embedding of the input embedding based on applying either the acoustic-visual feature extractor or the acoustic feature extractor to the acoustic information of the utterance included in the multimodal input. The system according to claim 1.

6. The one or more feature extractors include a text feature extractor, and the circuit is further configured to generate a second embedding of the input embedding based on applying the text feature extractor to the transcription of acoustic information related to the utterance and the transcription of different utterances that temporally precede or follow the utterance. The system according to claim 1.

7. The transformer encoder network includes a first stack of transformer encoders for a first modality of the multimodal input, a second stack of transformer encoders for a second modality of the multimodal input, and a third stack of transformer encoders for a third modality of the multimodal input. The system according to claim 1.

8. The network further comprises skip connections between each pair of adjacent transformer encoders within the transformer encoder network. The system according to claim 1.

9. The aforementioned circuit is Receiving one or more of the aforementioned videos, A scene detection model is applied to the one or more videos received, Based on the application of the scene detection model, multiple scenes are extracted from the one or more videos. A single boundary detection model is applied to each of the extracted scenes. Based on the application of the single boundary detection model, multiple utterances in the extracted multiple scenes are detected, Based on the above detection, a series of multimodal inputs are prepared. The multimodal inputs, which are further configured and input to the one or more feature extractors, are part of the prepared set of multimodal inputs. The system according to claim 1.

10. The fusion attention network includes one or more multi-head attention layers and a first fully coupled layer. The input of the first fully coupled layer is coupled to the output of the one or more multi-head attention layers. The system according to claim 1.

11. The circuit applies one or more multi-head attention layers to the emotion-related feature set, Determine the feature mapping within the aforementioned set of emotion-related features, Based on the feature mapping, the set of emotion-related features is linked to the latent representation of the set of emotion-related features. The system according to claim 10, further configured as follows.

12. The fused feature representation of the emotion-related feature set is further generated based on applying the first fully connected layer to the latent representation. The system according to claim 11.

13. The output network includes a second fully coupled layer coupled to the output of the fusion attention network. The system according to claim 1.

14. In a system that includes memory for storing a multimodal fusion network, The multimodal fusion network is configured to input multimodal inputs related to speech depicted in one or more videos into one or more feature extractors, including an acoustic-visual feature extractor and a visual feature extractor. The output of the one or more feature extractors for the input is to generate an input embedding that includes an embedding for each modality of the multimodal input, Based on applying the transformer encoder network of the multimodal fusion network to the input embedding, a set of emotion-related features is generated that includes one or more features corresponding to each modality of the multimodal input, Based on applying the fusion attention network of the multimodal fusion network to the emotion-related feature set, a fused feature representation of the emotion-related feature set is generated. The emotion label of the utterance is predicted based on applying the output network of the multimodal fusion network to the fused feature representation. Includes, The method further includes generating a third embedding of the input embedding based on applying the acoustic-visual feature extractor or the visual feature extractor to the facial information of one or more speaking characters in the frames of the one or more video, and to the scene information associated with the frames, The frame is a frame corresponding to the duration of the utterance in the one or more videos, Inputting one or more video frames corresponding to the duration of the utterance into the acoustic-visual feature extractor or the visual feature extractor, Based on applying the acoustic-visual feature extractor or the face detection model of the visual feature extractor to each of the received frames, one or more faces are detected in each of the received frames. To generate one or more bounding boxes that include the one or more faces detected, By applying either the acoustic-visual feature extractor or the visual feature extractor, the areas associated with each of the one or more bounding boxes are normalized. Based on the detected one or more faces and the normalization, a third embedding of the input embedding is generated as the output of the acoustic-visual feature extractor or the visual feature extractor. A method characterized by further comprising:

15. The multimodal input includes a first modality related to the acoustics of the utterance, a second modality related to the transcription of the utterance, and a third modality related to the visual aspects of the utterance. The method according to claim 14.

16. The fusion attention network includes one or more multi-head attention layers and a first fully coupled layer. The input of the first fully coupled layer is coupled to the output of the one or more multi-head attention layers. The method according to claim 14.

17. By applying one or more multi-head attention layers to the aforementioned set of emotion-related features, Determining the mapping between features within the aforementioned set of emotion-related features, Based on the feature mapping, the set of emotion-related features is linked to the latent representation of the set of emotion-related features, The further includes, the fused feature representation of the emotion-related feature set is generated based on applying the first fully connected layer to the latent representation, The method according to claim 16.

18. A non-temporary computer-readable medium storing computer-executable instructions that cause a circuit to perform an operation when executed by the circuit of the system, wherein the operation is: The multimodal fusion network involves inputting multimodal inputs related to speech depicted in one or more videos into one or more feature extractors, including an acoustic-visual feature extractor and a visual feature extractor. The output of the one or more feature extractors for the input is to generate an input embedding that includes an embedding for each modality of the multimodal input, Based on applying the transformer encoder network of the multimodal fusion network to the input embedding, a set of emotion-related features is generated that includes one or more features corresponding to each modality of the multimodal input, Based on applying the fusion attention network of the multimodal fusion network to the emotion-related feature set, a fused feature representation of the emotion-related feature set is generated. The emotion label of the utterance is predicted based on applying the output network of the multimodal fusion network to the fused feature representation. Includes, The aforementioned operation is, The method further includes generating a third embedding of the input embedding based on applying the acoustic-visual feature extractor or the visual feature extractor to the facial information of one or more speaking characters in the frames of the one or more video, and to the scene information associated with the frames, The frame is a frame corresponding to the duration of the utterance in the one or more videos, The aforementioned operation is, Inputting one or more video frames corresponding to the duration of the utterance into the acoustic-visual feature extractor or the visual feature extractor, Based on applying the acoustic-visual feature extractor or the face detection model of the visual feature extractor to each of the received frames, one or more faces are detected in each of the received frames. To generate one or more bounding boxes that include the one or more faces detected, By applying either the acoustic-visual feature extractor or the visual feature extractor, the areas associated with each of the one or more bounding boxes are normalized. Based on the detected one or more faces and the normalization, a third embedding of the input embedding is generated as the output of the acoustic-visual feature extractor or the visual feature extractor. A non-temporary computer-readable medium characterized by further including the following.

Images