A multi-modal sentiment analysis method and system

Abstract

The application discloses a multimodal emotion analysis method and system, first, for the multi-modal face video data to be analyzed, the pre-trained model is used to extract the data characteristics of different modes. Then, based on the idea of feature decoupling, the multi-modal features are projected to the mode sharing feature subspace and the mode private feature subspace to maximize the retention of information of different modes. In the feature decoupling module, the denoised multi-modal sharing features are extracted. For mode private feature learning, by minimizing the mutual information between each mode private representation, the private features of different modes are constrained to be independent of each other, and the multi-modal private features are extracted. Then, through a two-stage cross-modal attention fusion module, the fusion of different features is realized, and the multi-modal emotion analysis task is realized. The application provides accurate and efficient multi-modal emotion analysis service, which can bring convenient and timely psychological counseling service for psychological experts and consultants.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis methods, specifically a multimodal sentiment analysis method and system. Background Technology

[0002] Multimodal sentiment analysis, as an important research direction in natural language processing, computer vision, and speech signal processing, has received widespread attention in recent years. It aims to comprehensively analyze human emotional states by fusing data from different modalities such as text, audio, images, and video. This task has broad application prospects, including intelligent customer service, emotional robots, human-computer interaction, virtual assistants, and mental health analysis.

[0003] Early sentiment analysis research primarily focused on single text modalities, using data sources such as personal blogs, product reviews, movie comments, and forum posts. Natural language processing techniques were employed to learn and construct specific representations according to task requirements. However, single-text sentiment analysis techniques could not fully convey users' emotions to meet increasingly complex practical needs. In recent years, with the development of social networks and the improvement of computer processing power, multimedia data on the internet has experienced explosive growth, and the ways and carriers of expressing opinions have become increasingly diversified. For example, in product reviews, users often attach pictures to supplement their opinions, enhancing credibility and supporting their viewpoints. These pictures are often closely related to the text, providing supplementary information for sentiment analysis tasks. Multimodal technologies, by integrating information such as text, voice, facial dynamics, and body language, can provide more accurate and richer emotional cues, more clearly expressing metaphorical information. In the current internet and industrial environment, video, as a rich and complex multimedia data, contains multiple modalities such as images, audio, and text, each carrying its unique information.

[0004] While multimodal video data can provide richer feature information for sentiment analysis tasks, the heterogeneity between text, audio, and images poses challenges to model learning. Existing multimodal sentiment analysis research mainly revolves around multimodal representation fusion and representation learning. Representation fusion aims to fully utilize complementary information from different modalities to obtain an integrated, information-rich multimodal representation. Its key method is designing effective multimodal fusion methods, primarily feature fusion, decision fusion, and model fusion, which are currently hot topics in related research. Effective information fusion can further improve the performance of multimodal learning and sentiment analysis. The core objective of representation learning is to learn from individual monomodal data, hoping to obtain representations that are sufficiently relevant to the task. Multimodal data consists of a mixture of heterogeneous modalities; text content is typically expressed using discrete semantic symbols, while images and audio are generally multidimensional tensors and time series composed of real-valued arrays. Multimodal representation learning is mainly divided into joint representation learning and collaborative representation learning. Joint representation learning essentially merges the representation learning and fusion stages into a single process, while collaborative representation learning can be further divided into strong collaborative and weak collaborative learning. Representation learning is a crucial part of multimodal sentiment analysis, and the quality of representations has a significant impact on subsequent fusion and classification.

[0005] Multimodal sentiment analysis research has made significant progress. Combining the two research questions mentioned above, Das et al. proposed a classic multimodal sentiment analysis framework. In this framework, each modality learns its representation independently, and the representations are fused in the latter part of the model, allowing the model to fully learn complementary information between different modalities. However, existing research has shown that a good multimodal representation should include two aspects of information: modality-shared information and modality-private information. Shared information refers to the prominent features commonly exhibited by all modalities, emphasizing the commonalities between modalities. Modality-private information refers to information unique to a single modality, emphasizing the differences between different modalities. The classic framework mentioned above cannot learn modality-private features and cannot learn comprehensive multimodal features. Some studies only constrain the learning of the two types of representations through simple distance calculation functions, which is difficult to effectively distinguish between the two types of representations. Therefore, how to effectively learn multimodal representations and extract and integrate modality-shared and private information is a key issue worth exploring. Furthermore, existing research on multimodal sentiment analysis of video data has not considered the impact of noise inevitably introduced during recording and transmission on representation learning. Effectively separating noise components from representations is crucial for learning good representations. Based on this research background, this invention focuses on learning modal common features and modal private features, and addresses the problem of denoising modal features, proposing a multimodal sentiment analysis method. Summary of the Invention

[0006] To address the above problems, this invention proposes a multimodal sentiment analysis method and system.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A multimodal sentiment analysis method, characterized by the following steps:

[0009] (1) Multimodal feature extraction;

[0010] Feature extraction is performed on the input video dataset, extracting features from text, audio, and visual modalities respectively. Text vectors are extracted using the BERT model for the text modal, acoustic features, including pitch, loudness, and Mel frequency cepstral coefficients, are extracted using the COVAREP toolkit for the audio modal, and facial expression features, including facial contours and facial muscle motion features, are extracted using Openface for the visual modal. The extracted features are used for subsequent feature selection and fusion.

[0011] (2) Modal shared feature learning based on adversarial learning;

[0012] This module contains two adversarial parts: adversarial between the shared encoder and the modality discriminator, and adversarial between the shared encoder and the noise discriminator. First, the shared encoder projects pre-extracted text, acoustic and visual features into the modality shared subspace. The shared encoder parameters make the feature distribution similar, making it difficult for the modality discriminator to determine which modality the current feature comes from, thus realizing the learning of common features between modalities. At the same time, the shared encoder adversarially fights against the noise discriminator to improve the noise resistance of the shared encoder.

[0013] (3) Modal private feature learning based on mutual information;

[0014] This module calculates mutual information to constrain the feature distribution of different modalities, designs an independent encoder for each modality, minimizes the mutual information between private features of different modalities, and ensures that the projected private features have sufficient separation, thereby capturing unique information within the modality. In addition, it models the noise within the modality, generates Gaussian noise components, minimizes the mutual information between noise and private features, and maximizes the mutual information between private features and sentiment tags, so that private features can reduce the impact of noise and be closely related to the sentiment analysis task.

[0015] (4) Attention-based cross-modal feature fusion;

[0016] The cross-modal fusion module comprises two stages. The first stage involves cross-modal interaction, which uses a multi-head self-attention mechanism to exchange information between different modal representations, thereby enhancing the cross-modal representation. The second stage involves dynamic attention allocation, which utilizes an adaptive attention mechanism to dynamically adjust the weights based on the importance of each representation, ultimately obtaining the fused multimodal feature representation to complete the sentiment analysis task.

[0017] (5) Construct a multimodal sentiment analysis model;

[0018] Using the multimodal fusion feature vector output from step (4), the classification results of sentiment analysis are obtained using a fully connected layer;

[0019] (6) System function demonstration.

[0020] As a further improvement to the method of the present invention, the feature extraction module in step (1) extracts text, acoustic and visual multimodal features X from the video segment to be analyzed. m (m∈{t,a,v}), specifically, it uses a pre-trained BERT model to extract text vectors X. t The audio feature vector X was extracted using the COVAREP toolkit. a This paper uses OpenCV to extract static video frames, and Openface to extract facial information from the video frames, obtaining visual features X. v .

[0021] As a further improvement to the method of the present invention, the modal common feature learning module in step (2) performs adversarial learning with the modal discriminator and noise discriminator through a common encoder, thereby realizing the learning of modal common features and the improvement of noise resistance. Specifically, it includes the following steps:

[0022] (2-1) Construct a shared encoder to convert the text features X obtained in step (1) into a shared encoder. t Acoustic features X a and video features X v Embedded into the modal common feature subspace;

[0023] First, a one-dimensional convolutional neural network is used to map the textual, auditory, and visual modal features to a unified dimension, represented as:

[0024]

[0025] in k m The convolution kernel of the convolutional neural network is used, and then a bidirectional LSTM is used to map the modal features to a common feature space. The bidirectional LSTM preserves temporal features and can capture long-term dependencies in the sequence. The formula is:

[0026]

[0027] Where θ I These are shared parameters across the three modalities. This step utilizes the gating mechanism of LSTM to learn complex temporal dynamic information, thereby enhancing the representation capability of shared features.

[0028] (2-2) To ensure that the encoder learns the common representation cm It can conform to the feature distribution of modality sharing, and uses the idea of ​​generative adversarial networks to guide parameter learning. By constructing a modality discriminator and a shared encoder to form adversarial constraints, it guides the model to learn the shared features of modal ambiguity. The modality discriminator is constructed as follows:

[0029]

[0030] Among them, D m (·) is a modality discriminator used to determine modality tags. It is a weight matrix. Suppose there exists a modality discriminator. It has converged to the optimal solution, C m It is a common representation from mode m, and the mode label is determined by the mode discriminator. If Able to judge X very well m If the modality label, i.e., the result is close to 1, then it indicates that the current C... m It contains distinct modal features, making it completely distinguishable from other modes, and contains very little modal common information; therefore, C m The importance of modal common features and If they are inversely proportional, the importance of samples to modal-shared features is calculated as follows:

[0031]

[0032] (2-3) Explicitly model the noise components and construct a noise discriminator. Explicitly constrain the model to ignore components in the modal representation that approximate the noise distribution. Obtain the denoised multimodal representation. Introduce Gaussian noise vectors to optimize the shared representation learning ability. Using Gaussian noise vectors helps improve the network's ability to cope with noise in real data.

[0033]

[0034] in This represents a standard normal distribution with a central axis at 0 and a standard deviation of 1, where m∈{t,a,v}. Gaussian noise is then encoded using a common encoder to make it have a similar distribution.

[0035]

[0036] The generated Gaussian noise vector is encoded by a shared encoder to make its distribution approximate the modal shared subspace, and then processed by a noise discriminator. The input modal features are mapped to a probability distribution to determine whether the current feature belongs to the modal encoding or the generated noise vector. Adversarial learning between the noise discriminator and the encoder enhances the model's noise robustness. The generated common features should contain as many sentiment features as possible. The importance obtained from the previous modal discriminator is calculated together with that of the noise discriminator.

[0037]

[0038] Where y m The true modal label for the feature is specifically represented as y. t =[1,0,0], y a =[0,1,0], y v =[0,0,1], and the acquisition of multimodal common features for denoising is achieved through adversarial learning between the shared encoder and the modal discriminator and noise discriminator.

[0039] As a further improvement to the method of the present invention, the modal private feature learning module in step (3) minimizes the mutual information between modal private features and the mutual information between modal private features and noise components. In order to capture the characteristics within a modality, the three modal features are embedded into a private feature subspace through a private encoder, specifically as follows:

[0040] P t =S t (X t ;θ t (8)

[0041] P a =S a (X a ;θ a (9)

[0042] P v =S v (X v ;θ v (10)

[0043] Where t, a, and v represent text, acoustic, and visual modalities, respectively, and X t X a and X v It consists of pre-extracted multimodal features, with each modality having its own unique encoder S. m Each encoder learns the training parameters θ for the corresponding modality independently. m This yields the private representation P for each modality. m Where m∈{t,a,v};

[0044] Similar to the shared encoder, modal features are mapped using a one-dimensional convolutional network and a bidirectional LSTM.

[0045]

[0046]

[0047] in k m For the convolution kernel of a convolutional neural network,

[0048] (3-1) DiCyR is chosen to minimize the mutual information between noise and private modes. Cyclic reconstruction and gradient inversion are used to ensure that private mode features and noise components share very little information. The specific objective function is calculated as follows:

[0049]

[0050] Where P m These are modal-private features, where m∈{t,a,v} and m′≠m. and They are P m To P m′ and P m′ To P m The decoder, GRL(·), is the gradient inversion layer;

[0051] (3-2) Minimizing the mutual information between private features ensures that the private features of different modalities contain as much different information as possible. However, it does not consider the influence of noise unique to each modality. It is required to minimize the mutual information between the private features of a modality and its similarly distributed noise. Similar to modal common representation learning, the noise component is modeled using a Gaussian distribution. The generated noise distribution is passed through three private encoders:

[0052]

[0053] For m∈{t,a,v}, noise and private modal features generated with the same parameters will have the same distribution, similar to minimizing the private features. The objective function is as follows:

[0054]

[0055] Private encoders are capable of resisting noise and can distinguish between unique modal information and noise information.

[0056] As a further improvement to the method of the present invention, the cross-modal attention fusion module in step (4) performs information interaction through a multi-head self-attention mechanism and dynamically allocates weights according to the importance of each modal representation to improve the final sentiment analysis performance, specifically as follows:

[0057] (4-1) Information representations of each modality are enhanced through cross-modal attention mechanisms. Multi-head scaling dot product self-attention is used to learn the self-attention of the representations. Attention weights at each position in the input sequence are calculated in parallel and independently by multiple attention heads. Through steps (2) and (3), multimodal common representations and modal private representations F = {C, P} are learned. t ,P a ,P v}, where C represents the common feature of the three integrated modalities, C = C t +C a +C v By enhancing the source representation F through cross-modal attention, an enhanced representation F is obtained. cro The underlying process, let the initial reinforcement representation F cro ∈{C,P t ,P a ,P v First, F is normalized through a normalization layer. cro Embedded into a new feature space, while embedding the source representation F into two feature distributions.

[0058] Q cro =LN(F)W Q (17)

[0059] K cro =LN(F)W K (18)

[0060] V cro =LN(F)W V (19)

[0061] Among them, Q cro For updating the target representation weights, K cro and V cro The source character F is the normalized key and value. The embedding weights are the reinforcement representation and the source representation, respectively. LN is the layer normalization. Then, attention is used to perform cross-modal representation interaction, and the target representation is updated sequentially.

[0062]

[0063] The subsequent process of updating the representation is as follows:

[0064]

[0065] The final result

[0066] (4-2) It requires dynamically analyzing the importance of each reinforcement representation through an adaptive attention mechanism, assigning weights to it, and setting a shared attention vector. All modalities are updated together to obtain the attention value;

[0067]

[0068] in The attention values ​​are then normalized using the softmax function to obtain the final weights.

[0069]

[0070] The larger The corresponding representation is more important; the final representation is obtained through weighted summation.

[0071]

[0072] ⊙ represents weighted matrix multiplication, ultimately yielding the multimodal comprehensive representation F. fin To perform downstream sentiment analysis tasks.

[0073] As a further improvement to the method of the present invention, the multimodal sentiment analysis model described in step (5) uses the multimodal representation obtained in step (4) to identify the emotions of people in the video, and takes the sentiment label with the highest predicted probability score as the result of video analysis.

[0074] This invention discloses a system for multimodal sentiment analysis, comprising a user registration and login module, an online video conferencing module, a video recording and uploading module, and a sentiment analysis module. The user registration and login module facilitates user registration and login. The online video conferencing module assists users and psychological experts in online dialogue. The video recording and uploading module records or uploads user videos for subsequent sentiment analysis and visualization. The sentiment analysis module performs sentiment analysis on user-provided videos and visualizes the results for the user.

[0075] Beneficial effects: Compared with the prior art, the present invention adopts the above technical solution and has the following advantages:

[0076] (1) By adversarial interaction between the shared encoder and the modal discriminator, the learning of modal shared features is achieved, making the feature distributions of different modalities similar, improving the model's ability to extract common modal information, and promoting the effective integration of multimodal information. The adversarial interaction between the shared encoder and the noise discriminator improves the encoder's noise resistance, reduces the interference of noise on feature learning, and enhances the system's robustness in processing data in complex environments.

[0077] (2) By calculating mutual information, the mutual information of private features of different modalities is minimized, ensuring that the private features have a high degree of separation, which can more accurately capture the unique information within the modality and reduce the impact of modal noise on the sentiment analysis task.

[0078] (3) By using multi-head self-attention mechanism and dynamic attention allocation mechanism, the interaction and fusion of cross-modal features are enhanced, the importance of different modal features is dynamically adjusted, and finally high-quality multimodal feature representation is generated, thereby improving the accuracy of sentiment analysis. Attached Figure Description

[0079] Figure 1 This is the overall framework diagram of the present invention;

[0080] Figure 2 This is a schematic diagram of modality shared feature learning based on adversarial learning;

[0081] Figure 3 This is a schematic diagram of modal private feature learning based on mutual information;

[0082] Figure 4 This is a schematic diagram of attention-based cross-modal feature fusion;

[0083] Figure 5 This is the user registration and login display interface of the system of this invention;

[0084] Figure 6 This is the online video conferencing display interface of the system of this invention;

[0085] Figure 7 This is the video recording, uploading, and display interface of the system of this invention;

[0086] Figure 8 This is the interface for displaying the system sentiment analysis results of this invention. Detailed Implementation

[0087] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0088] The following is only one embodiment of the present invention. The present invention has many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention. All such corresponding changes and modifications should fall within the protection scope of the appended claims.

[0089] This invention provides a multimodal sentiment analysis method. First, features are extracted from data of different modalities (text, audio, and visual). Text features are extracted using the BERT model, audio features using COVAREP, and visual features using Openface to capture facial expressions. Then, these modal features are projected into a shared feature subspace and a private feature subspace. Adversarial learning is performed between a shared encoder and a modality discriminator and a noise discriminator to ensure that the shared features have a similar distribution and enhance noise resistance. To better learn the unique information of each modality, a mutual information minimization and maximization strategy is adopted to optimize the private encoder of each modality, ensuring that the private features are independent and closely related to the sentiment task. Simultaneously, an attention-based cross-modal feature fusion module is designed to dynamically allocate weights among different modalities, improving the accuracy of sentiment analysis.

[0090] like Figure 1 As shown, a multimodal sentiment analysis method of the present invention includes the following steps:

[0091] 1. Multimodal feature extraction

[0092] For the video data in the dataset, text, audio, and visual information are extracted and preprocessed. The text is embedded into a 768-dimensional vector using a pre-trained BERT model. The COVAREP toolkit is used to extract acoustic features from the audio data, including pitch, loudness, and Mel-frequency cepstral coefficients. For the video data, static keyframes are extracted, and Openface is used to extract facial expression features such as contours and facial muscle motion features.

[0093] 2. Modal shared feature learning based on adversarial learning

[0094] like Figure 2 As shown,

[0095] 1. Video Feature Extraction

[0096] To address the issue of inconsistent video frame rates in the ActivityNet Captions dataset, all videos are standardized to 25 frames per second. Videos with frame rates lower than 25 frames per second are padded, while those with frame rates higher than 25 frames per second undergo redundancy removal. The I3D model, pre-trained on the Sports-1M video dataset, is used to extract temporal features from the videos. The I3D network continuously reads video frames and outputs a fixed-length feature vector every 48 frames. The output of the seventh fully connected layer of the I3D network is dimensionality-reduced and embedded into a lower 500-dimensional vector space, which serves as the temporal feature representation of the video.

[0097] 2. Generation of Dense Video Summary Descriptions

[0098] (1) As Figure 2 As shown, based on the I3D features V={v1,v2,…,v3} of the video obtained in the previous step, T Using the bidirectional SST (Single-stream temporal) algorithm, employing both forward and reverse flow information, the probability of a given video segment exhibiting behavior is calculated at each time step. The overall confidence score C for each behavioral segment is then calculated. p The calculation formula is as follows:

[0099]

[0100] in This represents the confidence score for the positive flow. This represents the reverse flow confidence score. The calculation method is as follows:

[0101]

[0102] Where σ represents the Sigmoid function. This represents the hidden state of the LSTM cell at time step t. Output the final confidence score.

[0103] The set P of behavioral segments with scores above a threshold can be formally represented as:

[0104]

[0105] in Represents the behavior fragment p i The start time, Represents the behavior fragment p i End time, This indicates that the reverse-flow LSTM unit is in the behavior segment p i The hidden state at the start time step. This indicates that the forward-flowing LSTM unit is in the behavior segment p iThe hidden state at the end of the time step.

[0106] (2) For candidate behavior segments P = {p1, p2, ..., p M},like Figure 3 As shown, based on the pointer network, using the attention mechanism, a recurrent neural network outputs a probability distribution about the input sequence, and at each time step, a weight a is calculated about the set of candidate action segments P. t a t The calculation method is as follows:

[0107]

[0108] Where u(p) i It is calculated in the following way:

[0109] u(p i ) = [Loc(p i ),Vis(p i (5)

[0110] Vis(p i ) represents a behavior segment p i Visual features, Loc(p) i ) represents a behavior segment p i Its position within the entire video.

[0111] The weight a t As the probability of selecting each behavior segment, the input element with the highest probability is selected as the output, thereby achieving the goal of selecting a set of highly related behavior segments from the candidate behavior segments.

[0112] (3) Figure 4 As shown, a dynamic attention mechanism and a gating mechanism are used to fuse behavior category information, contextual visual information, and visual information of the current behavior segment, and a description is generated using the fused visual information. For the behavior segment... Based on its timestamp From the I3D feature sequence V = {v1, v2, ..., v T Extract p from} i Corresponding I3D feature set As visual information of the current behavioral segment.

[0113] Behavioral category information E is represented as follows:

[0114]

[0115] Where b represents the total number of behavior categories, Pr(l i The behavior category of the behavior segment is l. iThe probability of L is given by L, where L represents the number of video frames.

[0116] From the behavioral category feature sequence E = {e1, e2, ..., e L Extract p from} i The corresponding set of behavioral category features E i For E i Average pooling is performed to obtain the behavior category feature η. An LSTM is used as the decoder to generate the description. At time step t, the visual information input to the LSTM decoder can be formally represented as:

[0117]

[0118] The dynamic attention mechanism is used to fuse I3D features from the visual information of the current action segment. The calculation method is as follows:

[0119]

[0120] Representing I3D features v i+m-1 The corresponding weights are calculated as follows:

[0121]

[0122] Representing I3D features v i+m-1 The corresponding relevance score is calculated as follows:

[0123]

[0124] Among them, H t-1 This represents the hidden state of the decoder at time step t-1, where [·,·] denotes vector concatenation. Then, using a method similar to LSTM gating, the visual features are... It integrates visual features with contextual information. A fully connected layer is used to integrate visual features. The calculation formula is as follows: (This refers to the projection of contextual visual information into a space of the same dimension.)

[0125]

[0126] in and W ct It is a projection matrix, which can be learned automatically during training. Then, visual features are... With contextual visual features Fusion, yielding fusion feature F t (p i The calculation process is as follows:

[0127]

[0128] Where E t-1 H is the vector representation of the word output at time step t-1. r-1 This represents the hidden state of the decoder at time step t-1, where ⊙ represents the element-wise multiplication operation, and σ represents the Sigmoid activation function. The gating signal g... ct This controls the influence of the current behavioral fragment and its contextual visual information on description generation. Finally, the fused features F t (p i In the input decoder LSTM, at time step t, the probability distribution of each word in the vocabulary is calculated using the current hidden state. Based on the probability distribution, the word with the highest probability is selected as the output of the current time step.

[0129] The generated description is formally represented as: This represents a statement, where:

[0130] 1)M x Indicates the number of statements;

[0131] 2) and Representing statement S respectively i The start and end times of the corresponding video segment;

[0132] 3) w represents a word

[0133] 3. Behavior recognition and subject localization

[0134] The YOLOv3 object detection algorithm is used to detect people in the first k=16 frames of a video clip. A dependency parsing algorithm is then used to analyze descriptive sentences, extracting subjects and actions, and filtering person regions based on the subjects. The Deep Sort object tracking algorithm is used to locate the positions of each person in the video, and a Siamese network is used for correlation to reduce the chance of missing objects during tracking.

[0135] Table 1 shows the pseudocode for the candidate tracking and association algorithm.

[0136]

[0137] The behavior recognition part identifies human behavior based on the TSN (Temporal Segment Networks) algorithm. The video segment is divided into several equal-length segments. For each segment, a mask matrix is ​​used to focus the model on the region where the candidate is located. This can be formally represented as:

[0138]

[0139] Action recognition is performed using both spatial and temporal streams. For the spatial stream, the InceptionV3 convolutional neural network is used to extract image features of the regions containing candidate objects in video frames, and a Softmax classifier is used to obtain action category prediction scores. For the temporal stream, the OpenCV open-source toolkit is used to calculate optical flow and generate optical flow images. Five consecutive frames corresponding to the optical flow images are randomly selected from each small segment, and processed using a method similar to multi-channel convolution for RGB images to extract motion features. Then, a Softmax classifier is used to obtain action category prediction scores. The action category prediction score for each stream with respect to each small segment is calculated, and the prediction scores of all small segments are fused using the Top-K pooling method to obtain the action category prediction score for the entire video segment, which is formally represented as follows:

[0140]

[0141] in Let represent the predicted score of the i-th behavior category in the j-th segment, and N represent the total number of segments. When the j-th segment is selected, otherwise g i This represents the predicted score for the i-th behavior category in the entire video segment.

[0142] 4. Facial expression recognition of the actor

[0143] Based on the location information of the subject and the video segment information obtained in step (3), static image information, temporal information, and audio information are fused to perform facial expression recognition of the subject. Static image features and temporal features are obtained using the visual information of the video. Before extracting these two features, the video frame images need to be preprocessed, i.e., the region where the subject is located is determined based on the subject's location information, and then the face detection algorithm MTCNN (Multi-task convolutional neural network) is used to further locate the subject's facial region. For static image features, the VGG16 convolutional neural network is used to extract features of the facial region in the image frame, and the mean, maximum value, and standard deviation of all features are calculated and used as fused features. Based on the fused features, SVM is used for classification to obtain the facial expression category prediction score for the static image feature part.

[0144] For temporal features, a CNN-LSTM architecture is used for feature extraction, and the output of the LSTM at the last time step is used as the expression category prediction score for that part.

[0145] For audio features, a one-dimensional convolutional neural network consisting of seven convolutional layers and three pooling layers is used to extract audio features. Then, a softmax classifier is used for classification to obtain the expression category prediction score for the audio feature portion.

[0146] Finally, using a linear weighting method, weights are assigned to the expression category prediction scores obtained from the above three features, and then the weighted sums are used to obtain the final expression category prediction score.

[0147] 5. Construct a dense video description model that integrates facial expression information.

[0148] Based on the outputs of steps (2) and (4), the descriptive statements are parsed using a dependency parsing algorithm to extract sentence components. New descriptive statements are then generated using grammatical rules, resulting in a dense video description incorporating facial expression information. The generated results are as follows: Figure 5 As shown, the description can be formally represented as: single statement in:

[0149] (1)M x Indicates the number of statements;

[0150] (2) and Representing statement S respectively i The start and end times of the corresponding video segment;

[0151] (3) w represents a word;

[0152] (4)w k ∈{angry,disgust,fear,happy,network,sad,surprise}

[0153] 6. System function demonstration.

[0154] The system functionality demonstration includes displays of the data supplementation module, model performance comparison results, and description generation results. In the data supplementation module, users can upload videos and annotate them with corresponding reference descriptions to supplement the model training dataset. The model performance comparison results display provides performance analysis results of the model proposed in this patent compared with other relevant models. In the description generation interface, users select the video file to be described, click the upload button, and a description of the video content will be automatically generated and displayed on the description generation page, such as... Figure 6 As shown, the generated result includes a set of descriptive statements and corresponding timestamps.

[0155] This invention relates to a dense video description system integrating facial expression information, comprising a data management module, a data preprocessing module, a model training module, and a user interaction and visualization module. The data management module is primarily responsible for data supplementation and storage. Specifically, the data supplementation function adds data to the original dataset; the data storage function stores information needed for model training and the original dense video description dataset. In the data preprocessing module, the reference description text and video content in the dataset are preprocessed. For text, word segmentation and pause word removal are performed. For video, a pre-trained I3D model is used to represent it as an I3D feature set, and a pre-trained InceptionV3 model is used to extract its behavioral category features, mainly including three operations: video frame rate unification, dynamic feature extraction, and behavioral category feature extraction. In the model training module, the model is built and trained, including model parameter initialization, iterative input, and parameter updates. The user interaction and visualization module is primarily responsible for receiving and processing user requests and visualizing the description results. This system can solve the problem of insufficient information expression in existing dense video description research, and effectively improves the accuracy and richness of dense video description generation results.

[0156] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that: Includes the following steps: (1) Multimodal feature extraction; Feature extraction is performed on the input video dataset, extracting features from text, audio, and visual modalities respectively. Text vectors are extracted using the BERT model for the text modal, acoustic features, including pitch, loudness, and Mel frequency cepstral coefficients, are extracted using the COVAREP toolkit for the audio modal, and facial expression features, including facial contours and facial muscle motion features, are extracted using Openface for the visual modal. The extracted features are used for subsequent feature selection and fusion. (2) Modal shared feature learning based on adversarial learning; This module contains two adversarial parts: adversarial between the shared encoder and the modality discriminator, and adversarial between the shared encoder and the noise discriminator. First, the shared encoder projects pre-extracted text, acoustic and visual features into the modality shared subspace. The shared encoder parameters make the feature distribution similar, making it difficult for the modality discriminator to determine which modality the current feature comes from, thus realizing the learning of common features between modalities. At the same time, the shared encoder adversarially fights against the noise discriminator to improve the noise resistance of the shared encoder. (3) Modal private feature learning based on mutual information; This module calculates mutual information to constrain the feature distribution of different modalities, designs an independent encoder for each modality, minimizes the mutual information between private features of different modalities, and ensures that the projected private features have sufficient separation, thereby capturing unique information within the modality. In addition, it models the noise within the modality, generates Gaussian noise components, minimizes the mutual information between noise and private features, and maximizes the mutual information between private features and sentiment tags, so that private features can reduce the impact of noise and be closely related to the sentiment analysis task. (4) Attention-based cross-modal feature fusion; The cross-modal fusion module consists of two stages. The first stage is cross-modal interaction, which uses a multi-head self-attention mechanism to exchange information between different modal representations and enhance cross-modal representation. The second stage is dynamic attention allocation, which uses an adaptive attention mechanism to dynamically adjust the weights according to the importance of each representation, and finally obtains the fused multimodal feature representation to complete the sentiment analysis task. (5) Construct a multimodal sentiment analysis model; Using the multimodal fusion feature vector output from step (4), the classification results of sentiment analysis are obtained using a fully connected layer; (6) System function demonstration.

2. The multimodal sentiment analysis method according to claim 1, characterized in that: The feature extraction module described in step (1) extracts text, acoustic, and visual multimodal features X from the video segment to be analyzed. m (m∈{t,a,v}), specifically, it uses a pre-trained BERT model to extract text vectors X. t The audio feature vector X was extracted using the COVAREP toolkit. a This paper uses OpenCV to extract static video frames, and Openface to extract facial information from the video frames, obtaining visual features X. v .

3. The multimodal sentiment analysis method according to claim 1, characterized in that: Step (2) of the modal common feature learning module involves adversarial learning between the common encoder, modal discriminator, and noise discriminator to achieve the learning of modal common features and the improvement of noise resistance. Specifically, this includes the following steps: (2-1) Construct a shared encoder to convert the text features X obtained in step (1) into a shared encoder. t Acoustic features X a and video features X v Embedded into the modal common feature subspace; First, a one-dimensional convolutional neural network is used to map the textual, auditory, and visual modal features to a unified dimension, represented as: in k m The convolution kernel of the convolutional neural network is used, and then a bidirectional LSTM is used to map the modal features to a common feature space. The bidirectional LSTM preserves temporal features and can capture long-term dependencies in the sequence. The formula is: Where θ I These are shared parameters across the three modalities. This step utilizes the gating mechanism of LSTM to learn complex temporal dynamic information, thereby enhancing the representation capability of shared features. (2-2) To ensure that the encoder learns the common representation c m It can conform to the feature distribution of modality sharing, and uses the idea of ​​generative adversarial networks to guide parameter learning. By constructing a modality discriminator and a shared encoder to form adversarial constraints, it guides the model to learn the shared features of modal ambiguity. The modality discriminator is constructed as follows: Among them, D m (·) is a modality discriminator used to determine modality tags. It is a weight matrix. Suppose there exists a modality discriminator. It has converged to the optimal solution, C m It is a common representation from mode m, and the mode label is determined by the mode discriminator. If Able to judge X very well m If the modality label, i.e., the result is close to 1, then it indicates that the current C... m It contains distinct modal features, making it completely distinguishable from other modes, and contains very little modal common information; therefore, C m The importance of modal common features and If they are inversely proportional, the importance of samples to modal-shared features is calculated as follows: (2-3) Explicitly model the noise components and construct a noise discriminator. Explicitly constrain the model to ignore components in the modal representation that approximate the noise distribution. Obtain the denoised multimodal representation. Introduce Gaussian noise vectors to optimize the shared representation learning ability. Using Gaussian noise vectors helps improve the network's ability to cope with noise in real data. in This represents a standard normal distribution with a central axis at 0 and a standard deviation of 1, where m∈{t,a,v}. Gaussian noise is then encoded using a common encoder to make it have a similar distribution. The generated Gaussian noise vector is encoded by a shared encoder to make its distribution approximate the modal shared subspace, and then processed by a noise discriminator. The input modal features are mapped to a probability distribution to determine whether the current feature belongs to the modal encoding or the generated noise vector. Adversarial learning between the noise discriminator and the encoder enhances the model's noise robustness. The generated common features should contain as many sentiment features as possible. The importance obtained from the previous modal discriminator is calculated together with that of the noise discriminator. Where y m The true modal label for the feature is specifically represented as y. t =[1,0,0], y a =[0,1,0], y v =[0,0,1], and the acquisition of multimodal common features for denoising is achieved through adversarial learning between the shared encoder and the modal discriminator and noise discriminator.

4. The multimodal sentiment analysis method according to claim 1, characterized in that: In step (3), the modal private feature learning module minimizes the mutual information between modal private features and the mutual information between modal private features and noise components. In order to capture the characteristics within a modality, the three modal features are embedded into a private feature subspace through a private encoder, specifically as follows: P.S t sS t (X t θ t ) (8) P.S a sS a (X a θ a ) (9) P.S v sS v (X v θ v ) (10) Where t, a, and v represent text, acoustic, and visual modalities, respectively, and X t X a and X v It consists of pre-extracted multimodal features, with each modality having its own unique encoder S. m Each encoder learns the training parameters θ for the corresponding modality independently. m This yields the private representation P for each modality. m Where m∈{t,a,v}; Similar to the shared encoder, modal features are mapped using a one-dimensional convolutional network and a bidirectional LSTM. in k m For the convolution kernel of a convolutional neural network, (3-1) DiCyR is chosen to minimize the mutual information between noise and private modes. Cyclic reconstruction and gradient inversion are used to make the private features of modes and noise components share very little information. The specific objective function is calculated as follows: Where P m These are modal-private features, where m∈{t,a,v} and m′≠m. and They are P m To P m′ and P m′ To P m The decoder, GRL(·), is the gradient inversion layer; (3-2) Minimizing the mutual information between private features ensures that the private features of different modalities contain as much different information as possible. However, it does not consider the influence of noise unique to each modality. It is required to minimize the mutual information between the private features of a modality and its similarly distributed noise. Similar to modal common representation learning, the noise component is modeled using a Gaussian distribution. The generated noise distribution is passed through three private encoders: For m∈{t,a,v}, noise and private modal features generated with the same parameters will have the same distribution, similar to minimizing the private features. The objective function is as follows: Private encoders are capable of resisting noise and can distinguish between unique modal information and noise information.

5. The multimodal sentiment analysis method according to claim 1, characterized in that: The cross-modal attention fusion module in step (4) interacts with information through a multi-head self-attention mechanism and dynamically assigns weights based on the importance of each modal representation to improve the final sentiment analysis performance, specifically as follows: (4-1) Information representations of each modality are enhanced through cross-modal attention mechanisms. Multi-head scaling dot product self-attention is used to learn the self-attention of the representations. Attention weights at each position in the input sequence are calculated in parallel and independently by multiple attention heads. Through steps (2) and (3), multimodal common representations and modal private representations F = {C, P} are learned. t ,P a ,P v }, where C represents the common feature of the three integrated modalities, C = C t +C a +C v By enhancing the source representation F through cross-modal attention, an enhanced representation F is obtained. cro The underlying process, let the initial reinforcement representation F cro ∈{C,P t ,P a ,P v First, F is normalized through a normalization layer. cro Embedded into a new feature space, while embedding the source representation F into two feature distributions. Q cro =LN(F)W Q (17) K cro =LN(F)W K (18) V cro =LN(F)W V (19) Among them, Q cro For updating the target representation weights, K cro and V cro The source character F represents the normalized keys and values. The embedding weights are the reinforcement representation and the source representation, respectively. LN is the layer normalization. Then, attention is used to perform cross-modal representation interaction, and the target representation is updated sequentially. The subsequent process of updating the representation is as follows: The final result (4-2) It requires dynamically analyzing the importance of each reinforcement representation through an adaptive attention mechanism, assigning weights to it, and setting a shared attention vector. All modalities are updated together to obtain the attention value; in The attention values ​​are then normalized using the softmax function to obtain the final weights. The larger The corresponding representation is more important; the final representation is obtained through weighted summation. ⊙ represents weighted matrix multiplication, ultimately yielding the multimodal comprehensive representation F. fin To perform downstream sentiment analysis tasks.

6. The multimodal sentiment analysis method and system according to claim 1, characterized in that: The multimodal sentiment analysis model described in step (5) uses the multimodal representation obtained in step (4) to identify the emotions of people in the video, and takes the sentiment label with the highest predicted probability score as the result of video analysis.

7. A system for a multimodal sentiment analysis method according to any one of claims 1-6, characterized in that: The system includes a user registration and login module, an online video conferencing module, a video recording and uploading module, and a sentiment analysis module. The user registration and login module allows users to register and log in. The online video conferencing module assists users and psychological experts in conducting online dialogues. The video recording and uploading module records or uploads user videos for subsequent modules to perform sentiment analysis and visualize the results. The sentiment analysis module performs sentiment analysis on user-provided videos and visualizes the results for the user.

Images