System for multimodal multitasking learning for group cohesiveness and group emotion prediction

Abstract

The present invention relates to a multimodal multitasking learning system for predicting group cohesion and group emotion, and more specifically, to a multimodal multitasking learning system for predicting group cohesion and group emotion for extracting audio and visual features from a video. According to the present invention, the system comprises an extraction unit that extracts video frames and voice from a group-related video, a task unit that performs a task of predicting the cohesion and emotion of the group based on visual features extracted from the video frames and voice features extracted from the voice, and a learning unit that performs learning based on the loss resulting from the execution of the task, thereby having the effect of accurately predicting not only the level of group cohesion but also the emotional state of the group appearing in the video.

Description

Technology Field

[0001] The present invention relates to a multimodal multitasking learning system for group cohesion and group emotion prediction, and more specifically, to a multimodal multitasking learning system for group cohesion and group emotion prediction for extracting audio and visual features from video. Background Technology

[0002] Through mutual interaction and social connections, people can satisfy their needs for survival and prosperity. Therefore, individuals form and maintain relationships with one another to achieve their respective needs and goals. Group cohesion can be defined as the degree to which group members can work together to achieve common goals. Group emotion is a term that describes the emotions, attitudes, and dispositions shared by group members.

[0003] It can be viewed as an emotional entity exerting a top-down influence on the emotional states of individual members, or as a collection of the members' emotional states. Consequently, the social and emotional interactions among group members play a role in strengthening the bonds between them.

[0004] Therefore, research on group emotions and group cohesion is important. Knowledge of the dynamics of group cohesion can be greatly aided by the perception of emotions shared by group members, making it one of the most important research topics in the field of affective computing. Prior art literature

[0005] Published Patent 2022-0063816 (May 18, 2022) The problem to be solved

[0006] The present invention aims to solve the aforementioned problem by providing a multimodal multitasking learning system for group cohesion and group emotion prediction for extracting audio and visual features from video. means of solving the problem

[0007] To achieve the above objective, the present invention includes an extraction unit that extracts video frames and voice from a group-related video, a task unit that performs a task of predicting the cohesion and emotion of the group based on visual features extracted from the video frames and voice features extracted from the voice, and a learning unit that performs learning based on the loss resulting from the execution of the task.

[0008] The extraction unit separates the video's visual frames and audio waveforms before extracting features from the video recording; to extract visual features, it crops face images from each frame using a face extractor and extracts facial attributes and expressions; and to extract audio features, it extracts Arousal, Valence, and Dominance features to integrate the emotional characteristics of language.

[0009] The task unit performs a task of predicting the cohesion and emotion of a group based on visual features extracted from video frames and voice features extracted from voice, and includes an encoding module that inputs into a visual time encoder, compresses into a global average pooling layer, generates feature vectors for face attributes and facial expressions through the time dimension, and performs encoding; an MHA fusion module that fuses the representations of the feature vectors to concretize visual and auditory representations into fused feature vectors; and a prediction module that inputs the fused feature vectors into a fully connected layer to predict the emotional state and cohesion level.

[0010] A multimodal multitasking learning system for group cohesion and group emotion prediction, characterized by using a loss function combined with two tasks of group cohesion and group emotion prediction, and using mean squared error loss to minimize the difference between group cohesion and group emotion prediction.

[0011] Loss function L :

[0012]

[0013] Here, N is the number of samples, and is the actual group cohesion and predicted cohesion score, is the predicted probability of emotional state, and is a focal element. Effects of the invention

[0014] According to the present invention, not only the level of cohesion of a group appearing in a video but also its emotional state can be accurately predicted. Furthermore, the present invention can be widely utilized in various fields / locations (e.g., companies, schools, etc.) for purposes such as monitoring groups of interest. Brief explanation of the drawing

[0015] FIG. 1 is a configuration diagram showing a multimodal multitasking learning system for predicting group cohesion and group emotion according to an embodiment of the present invention, and Figure 2 is a basic architecture for predicting group cohesion and group sentiment. Figure 3 shows an MHA fusion module for scenarios including dual-mode and triple-mode. Specific details for implementing the invention

[0016] The present invention will be described in detail below with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by exemplary embodiments. Identical reference numerals in each drawing indicate components that perform substantially the same function.

[0017] The purpose and effects of the present invention may be naturally understood or become clearer through the following description, and the purpose and effects of the present invention are not limited solely to the description below. Furthermore, in describing the present invention, if it is determined that a detailed description of known technology related to the present invention may unnecessarily obscure the essence of the present invention, such detailed description will be omitted.

[0018] The present invention relates to a system for predicting group cohesion and emotion, comprising a fused deep learning technology for extracting audio and visual features from a video, and an emotion recognition model that extracts and fuses audio and visual features of data to simultaneously predict the level of group cohesion and the emotional state of the group based on the conversation of members within the video. Accordingly, the present invention is characterized by extracting video frames and voice from a group-related video, performing a task to predict the cohesion and emotion of the group based on various visual features extracted from the video frames and various voice features extracted from the voice, and performing learning based on the loss resulting from the execution of the task.

[0019] That is, the present invention performs multi-task (cohesion prediction + sentiment prediction) based learning based on multimodal features (=visual features + speech features).

[0020] FIG. 1 is a configuration diagram showing a multimodal multitasking learning system for predicting group cohesion and group emotion according to one embodiment of the present invention, and FIG. 2 is a basic architecture for predicting group cohesion and group emotion.

[0021] As illustrated in FIG. 1, the multimodal multitasking learning system (10) for predicting group cohesion and group emotion according to the present embodiment includes an extraction unit (100), a task unit (200), and a learning unit (300).

[0022] The extraction unit (100) is configured to extract video frames and audio from a group-related video. This extraction unit performs feature extraction, and before extracting features from the video recording, it first separates the visual frames of the video and the waveforms of the audio.

[0023] To begin the process of extracting visual features, a face extractor is used to crop the face photos from each frame. Then, two distinct types of expressions—face attributes and facial expressions—are extracted.

[0024] A pre-trained model is used to generate face feature vectors. The average of all face attributes in a single frame is calculated, and then frames are sampled from the entire video using a window slide.

[0025] Finally, encoding function sequences are collected for each facial attribute and facial emotion. In addition to facial feature extraction, a comprehensive representation of the video content is extracted.

[0026] Audio features extract 88 characteristics useful for speech and emotion computing.

[0027] Speech features (Prosodic, excitation, vocal tract, spectral, and cepstral MFCCs) are all integrated into GeMAPS, and the sub-MFCC (Mel-frequency cepstrum) is also included.

[0028] To integrate the emotional characteristics of language, Arousal, Valence, and Dominance features are extracted. 1024 features are collected from each audio sample after the head layer is ignored in the process. Additionally, the audio waveform is transformed into a log Mel spectrogram with 128 Mel bands, a Fast Fourier Transform (FFT) window of length 2048, 512 samples between frames, and a maximum frequency of 8 kHz. The final image size that the spectrogram can accommodate is 128x1292.

[0029] The task unit (200) is configured to perform a task of predicting the cohesion and emotion of a group based on various visual features extracted from video frames and various voice features extracted from voice. This task unit (200) includes an encoding module (210), an MHA fusion module (220), and a prediction module (230).

[0030] The encoding module (210) performs the task of predicting the cohesion and emotion of the group based on visual features extracted from video frames and voice features extracted from voice, and then inputs into a visual time encoder and compresses into a global average pooling layer, and then generates feature vectors for face attributes and facial expressions through the time dimension to perform encoding.

[0031] The MHA fusion module (220) fuses the representations of feature vectors to concretize visual and auditory representations into fused feature vectors. At this time, the concretization is concretized into three layers of input attention, cross attention, and self-attention that constitute the MHA fusion module, which will be explained below.

[0032] The prediction module (230) inputs the fusion feature vector of the MHA fusion module (220) into a fully connected layer to predict the emotional state and the level of cohesion.

[0033] The task unit according to the present embodiment is a Multi-task Multi-Feature Fusion Framework.

[0034] The general architecture of a baseline model for predicting group cohesion and group emotion through multi-feature fusion is shown in Fig. 2. In the first step of the process, each of the two sequences of facial expressions is input into a visual time encoder to perform the following, utilizing information through the time dimension.

[0035] The temporal dimension is compressed using a global mean pooling layer, and then 100 feature vectors for face attributes and 200 feature vectors for face expressions are obtained.

[0036] To compress the feature dimensions of each of these representations, a fully connected layer is applied to the feature representations extracted from SlowFast, OpenSMILE, and Wave2vec. In summary, 100 feature vectors are generated for SlowFast and Wave2vec representations, but only 32 feature vectors are generated for the OpenSMILE representation because the input size is smaller.

[0037] Next, the log-melspectrogram image is encoded using a Squeeze-and-Excitation (SE) network.

[0038] In addition, a fusion module is constructed using the Multi-Head Attention (MHA) approach. The three layers constituting the MHA fusion module are Input Attention, Cors Attention, and Self-Attention.

[0039] MHA requires a set of queries, keys, and values ​​as inputs to be transformed, using a linear projection that independently learns h (number of heads). Before fusing these representations, it uses input attention levels to align various features within the same domain.

[0040] Cross-Attention is used to blend the functions of paired modalities, where one modality facilitates the generation of an attraction map for another modality. If M is the number of modalities, then M(M-1) is the number of cross-attention blocks.

[0041] Then, Self-Attention combines cross-modal data for each modality to refine the representation.

[0042] Figure 3 shows an MHA fusion module for scenarios including by-modality and tri-modality. In Figure 3, a) is the by-modality case in the MHA fusion module, and b) is the tri-modality case. To use MHA fusion, first, a tri-modality MHA fusion module is used for each of the three visual and auditory representations, and fused characteristics for each branch are obtained.

[0043] Then, the next bimodal MHA fusion module is used for this fusion function, and the resulting fusion feature vector is input into the fully connected layer to predict emotional states and cohesion levels.

[0044] The learning unit (300) is configured to perform learning based on the loss resulting from the execution of a task. This learning unit uses a loss function combined with two tasks, group cohesion and group sentiment prediction, to optimize model convergence. Mean squared error loss is used to minimize the difference between the level of group cohesion and the model prediction. A loss function focused on the classification problem is used due to class imbalance, and the final loss function L can be expressed as follows.

[0045]

[0046] Here, N is the number of samples, and is the actual group cohesion and predicted cohesion score, is the predicted probability of emotional state, and is a focal element.

[0047] Numerous studies based on findings in social science and psychology aim to identify and classify the dynamics among members of specific groups. Group cohesion plays a crucial role in maintaining and organizing a group, as it encompasses both the motivation and sense of purpose of its members.

[0048] In this invention, a multimodal multitasking system is proposed to predict group cohesion and group emotion by utilizing an artificial intelligence deep learning model.

[0049] First, the visual data extracted three types of expression regarding facial features, facial emotions, and the overall video environment.

[0050] In addition, three types of audio features were extracted using OpenSMILE, Wave2vec 2.0, and spectrogram encoding.

[0051] Then, the MHA fusion module is used to combine these feature representations, and the fusion output is input into a fully connected layer to predict both the group cohesion score and the group sentiment probability. Based on these predictions, it is possible to better understand how strong the cohesion of group members is.

[0052] Predicting and understanding the cohesion of group members can help organizational managers lead the organization efficiently. Individuals belonging to highly cohesive groups can have a significant impact on business performance and make a major contribution to achieving organizational results.

[0053] Individuals who have difficulty living as members of society engage in group activities through group psychological counseling, and group cohesion is assessed before and after the counseling to verify its effectiveness. In this case, by using the group cohesion prediction AI model developed in this invention, group cohesion can be objectively and accurately determined through captured video.

[0054] For group members, emotions, as well as cohesion, play a crucial role in maintaining the group. Identifying and analyzing the emotions of group members makes it possible to determine the triggers that cause changes in their feelings, which can serve as an important indicator of the group's stability.

Claims

Claim 1 It includes an extraction unit that extracts video frames and audio from group-related videos, a task unit that performs a task of predicting the cohesion and emotion of the group based on visual features extracted from video frames and audio features extracted from audio, and a learning unit that performs learning based on the loss resulting from the execution of the task. The task unit generates face attribute feature vectors and face expression feature vectors by including a visual time encoder and a global mean pooling layer, and encodes Arousal, Valence, and Dominance (AVD) features together with Mel Spectrogram-based acoustic features to integrate the emotional characteristics of language, and sequentially fuses visual and auditory features using a Multi-Head Attention (MHA) fusion module composed of Input Attention, Cross Attention, and Self Attention, and predicts group cohesion and emotion using the fused features. The learning unit uses Mean Squared Error (MSE) loss for cohesion prediction and Focal Loss for emotion prediction to minimize the difference between the level of group cohesion and the model prediction. A multimodal multitasking learning system for group cohesion and group emotion prediction characterized by performing learning based on a combined loss function. Claim 2 A multimodal multitasking learning system for predicting group cohesion and group emotion according to claim 1, wherein the extraction unit separates the visual frames of the video and the waveforms of the audio before extracting features from the video recording, cuts out face photos from each frame using a face extractor to extract visual features, then extracts facial attributes and expressions of facial expressions, and extracts features of arousal, valence, and dominance to integrate the emotional characteristics of the language to extract audio features. Claim 3 A multimodal multitasking learning system for predicting group cohesion and group emotion according to claim 1, wherein the task unit performs a task of predicting the cohesion and emotion of a group based on visual features extracted from video frames and voice features extracted from voice, and comprises: an encoding module that inputs to a visual time encoder, compresses to a global average pooling layer, generates feature vectors for face attributes and feature vectors for face expressions through the time dimension, and performs encoding; an MHA fusion module that fuses the expressions of the feature vectors to concretize visual and auditory expressions into fused feature vectors; and a prediction module that inputs the fused feature vectors into a fully connected layer to predict the emotional state and cohesion level. Claim 4 A multimodal multitasking learning system for group cohesion and group emotion prediction according to claim 1, wherein the learning unit uses a loss function combined with two tasks of group cohesion and group emotion prediction, and uses mean squared error loss to minimize the difference between group cohesion and group emotion prediction. Loss function L : Here, N is the number of samples, and is the actual group cohesion and predicted cohesion score, is the predicted probability of emotional state, and is a focal element.

Images