A multi-modal sentiment analysis method and system based on large model video description

Abstract

The application discloses a kind of multi-modal sentiment analysis method and system based on large model video description, method includes: obtaining sound video segment, sound video segment is decomposed into video modal data, audio modal data and text modal data;Video modal data, audio modal data and text modal data are carried out single modal embedding processing, obtain video modal feature vector, audio modal feature vector and text modal feature vector;According to video modal feature vector and audio modal feature vector, text modal feature vector is carried out information enhancement processing, and enhanced text modal feature vector is obtained;Video modal feature vector, enhanced text modal feature vector and audio modal feature vector are carried out cross-modal fusion processing in large model block, and sentiment analysis output value is obtained.The embodiment of the application can carry out sentiment analysis in combination with more information in original multi-modal data, and the analysis result is more accurate, and can be widely applied in computer technology field.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a multimodal sentiment analysis method and system based on large-model video description. Background Technology

[0002] With the rapid development of social networks, an increasing number of videos containing personal emotional information are being posted on social platforms. These videos contain rich visual and auditory information, and often also provide a large amount of textual information containing personal opinions, making them natural multimodal data. This provides substantial data support for sentiment analysis in multimodal scenarios. Multimodal sentiment analysis uses information contained in multiple modal signals simultaneously and analyzes it through models to summarize the emotions contained in the multimodal data. Early sentiment analysis mainly analyzed emotions by considering the relationship between phrases and sentences; however, sarcastic or mocking emotions often require consideration of tone of voice and facial expressions for accurate identification.

[0003] Current mainstream multimodal sentiment analysis methods decompose audio videos into three modalities: video, audio, and text. Neural networks then extract information from these three modalities to analyze human emotions. Current research in this area can be divided into three main categories. The first category is early fusion, such as Multiple Kernel Learning (MKL), which typically occurs during the input phase and has a simple framework. The second category is late fusion, where models often contain multiple branches. This structure facilitates encoding modality-specific information within a single modality, ultimately combining the results from different branches during the decision-making phase. Both early and late fusion methods have drawbacks: 1) Early fusion methods combine different modalities too early, lacking the encoding process within a single modality. This leads to performance issues in cross-modal modeling due to the lack of contextual information within a single modality. 2) Late fusion methods have separate models for different modalities, interacting only during the decision-making phase. This prevents effective information complementarity between modalities, especially in samples with modal conflicts, often resulting in incorrect judgments. The third type of approach is hybrid fusion, which addresses the shortcomings of early and late fusion methods and combines their advantages. These include Tensor Fusion Networks (TFN), Multi-Attention Recurrent Networks (MARN), and Hybrid Contrastive Learning Networks (HyCon). TFN uses LSTMs with forget gates to model temporally relevant information and then models interactions between multiple modalities through Cartesian product. This method effectively preserves modality-specific information, but the tensors obtained through Cartesian product often have high dimensionality, increasing the risk of overfitting. HyCon designs three different contrastive learning methods to bridge the gaps between different modalities and samples of the same modality but different classes in a unified feature space, effectively eliminating the influence of modal heterogeneity. It then fuses these modalities through an attention mechanism. While this fusion method can ignore the effects of modality misalignment, cross-modal attention has high time complexity and consumes significant computational resources. Furthermore, the text modal inputs of these models only contain dialogue content between characters in the video, without involving other information such as background changes or character states. This limits the fusion capabilities centered on the text modality, and the prediction accuracy of the models needs to be improved. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in related technologies. To this end, this invention proposes a highly accurate and efficient multimodal situation analysis method and system based on large-model video description.

[0005] On one hand, embodiments of the present invention provide a multimodal sentiment analysis method based on large-model video description, including:

[0006] Acquire a video segment with audio, and decompose the video segment with audio into video modal data, audio modal data, and text modal data;

[0007] The video modal data, the audio modal data, and the text modal data are subjected to single-modal embedding processing to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors;

[0008] Information enhancement processing is performed on the text modality feature vector based on the video modality feature vector and the audio modality feature vector to obtain the enhanced text modality feature vector;

[0009] The video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector are subjected to cross-modal fusion processing in a large model block to obtain the sentiment analysis output value.

[0010] Optionally, the step of acquiring the audio video segment and decomposing the audio video segment into video modal data, audio modal data, and text modal data includes:

[0011] Extract several keyframes from the video clip with sound;

[0012] The video frames corresponding to the keyframes are used as video modal data;

[0013] The audio corresponding to the keyframe is used as audio modal data;

[0014] Video content detection and speech recognition are performed on the video frames and audio modal data to obtain text modal data.

[0015] Optionally, the step of performing single-modal embedding processing on the video modal data, the audio modal data, and the text modal data to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors includes:

[0016] The video modal data is encoded from pixels to feature vectors to obtain a video sequence vector;

[0017] Generate a word vector sequence based on the text modality data;

[0018] Audio features are extracted from the audio modal data to obtain an audio sequence vector;

[0019] The video sequence vector, the word vector sequence, and the audio sequence vector are subjected to temporal information aggregation processing to obtain video modal feature vector, text modal feature vector, and audio modal feature vector.

[0020] Optionally, the step of performing temporal information aggregation processing on the video sequence vector, the word vector sequence, and the audio sequence vector to obtain video modal feature vectors, text modal feature vectors, and audio modal feature vectors includes:

[0021] Calculate the cosine similarity between the video sequence vector, the word vector sequence, and the audio sequence vector;

[0022] Based on the cosine similarity, the feature vectors of several time steps with the highest information content are determined, resulting in video modal feature vectors, text modal feature vectors, and audio modal feature vectors.

[0023] Optionally, the step of performing information enhancement processing on the text modal feature vector based on the video modal feature vector and the audio modal feature vector to obtain an enhanced text modal feature vector includes:

[0024] The learnable scalar is determined based on the feature vectors of each modality;

[0025] Configure the first parameter matrix, and determine the first intermediate vector based on the feature vectors of each modality and the first parameter matrix;

[0026] Calculate the information value of each mode combination based on the first intermediate vector and the learnable scalar;

[0027] Information values ​​are integrated to obtain an enhanced text modality feature vector.

[0028] Optionally, the cross-modal fusion processing of the video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector in a large model block to obtain the sentiment analysis output value includes:

[0029] The video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector are mapped into a common projection space;

[0030] Configure the second parameter matrix, and combine the second parameter matrix to perform cross-modal fusion vectors on the mapped modal feature vectors;

[0031] The cross-modal fusion vector is subjected to linear and nonlinear transformations to obtain the sentiment analysis output value.

[0032] Optionally, the method further includes:

[0033] A multimodal sentiment analysis model is constructed, and the multimodal sentiment analysis model is trained to obtain a target multimodal sentiment analysis model; wherein, the target multimodal sentiment analysis model is used to perform sentiment analysis on the audio video segment.

[0034] On the other hand, embodiments of the present invention provide a multimodal sentiment analysis system based on large-model video description, including:

[0035] The first module is used to acquire audio video segments and decompose the audio video segments into video modal data, audio modal data and text modal data;

[0036] The second module is used to perform single-modal embedding processing on the video modal data, the audio modal data, and the text modal data to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors;

[0037] The third module is used to perform information enhancement processing on the text modal feature vector based on the video modal feature vector and the audio modal feature vector to obtain an enhanced text modal feature vector;

[0038] The fourth module is used to perform cross-modal fusion processing on the video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector in the large model block to obtain the sentiment analysis output value.

[0039] It should be noted that, in some embodiments, the system may further include the following modules:

[0040] The fifth module is used to construct a multimodal sentiment analysis model, train the multimodal sentiment analysis model to obtain a target multimodal sentiment analysis model; wherein, the target multimodal sentiment analysis model is used to perform sentiment analysis on the audio video segment.

[0041] On the other hand, embodiments of the present invention also provide an electronic device, including: a processor and a memory; the memory is used to store a program; the processor executes the program to implement the method described above.

[0042] On the other hand, embodiments of the present invention also provide a computer storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the method described above.

[0043] The embodiments of the present invention have the following beneficial effects: The embodiments of the present invention acquire audio video segments, decompose the audio video segments into video modal data, audio modal data, and text modal data; perform single-modal embedding processing on the video modal data, audio modal data, and text modal data to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors; perform information enhancement processing on the text modal feature vectors based on the video modal feature vectors and audio modal feature vectors to obtain enhanced text modal feature vectors; and perform cross-modal fusion processing on the video modal feature vectors, enhanced text modal feature vectors, and audio modal feature vectors in a large model block to obtain the sentiment analysis output value. This overall process can combine more information such as background changes, tone of voice, and state of voice contained in the video and audio information of the original multimodal data for sentiment analysis, resulting in more accurate analysis results that are more in line with actual application scenarios. Attached Figure Description

[0044] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0045] Figure 1 This is a flowchart illustrating the steps of the multimodal sentiment analysis method based on large model video description provided in this embodiment of the invention.

[0046] Figure 2 This is a flowchart of the multimodal sentiment analysis method based on large model video description provided in the embodiments of the present invention;

[0047] Figure 3 This is a process diagram of each stage of the multimodal sentiment analysis method based on large model video description provided in the embodiments of the present invention;

[0048] Figure 4 This is a schematic diagram of the information enhancement processing flow provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the cross-modal fusion processing flow provided in an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of a multimodal sentiment analysis system based on large model video description provided in an embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0053] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100," "second / S200," etc., in the specification, claims, and the aforementioned figures are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0054] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] A video clip can be decomposed into three modalities: video, audio, and text. This invention uses separate encoders for embedding to obtain visual modal data V, audio modal data A, and text modal data T, where... and The visual modality data and audio modality data are respectively composed of vectors from different time steps of the corresponding modality, L v and L a These represent the total number of time steps for the video and audio, respectively. The text modal data represents multiple word vectors, L t The fused feature represents the word count of the text and is obtained by fusing the three modalities. It can be used for multimodal sentiment analysis.

[0056] First, some technical terms involved in the embodiments of the present invention will be introduced:

[0057] FFmpeg: FFmpeg is an open-source computer program that can be used to record, convert, and stream digital audio and video. In this embodiment of the invention, it is used to extract keyframes of multimodal data.

[0058] YOLOv8: YOLOv8 is a target detection algorithm in the YOLO series, which can detect the position and category of multiple objects in an image or video in real time. In this embodiment of the invention, it is used to obtain the position of a person in a video.

[0059] FaceNet: FaceNet is an algorithm for face recognition, which is used in this invention for the initial extraction of face images.

[0060] Whisper: Whisper is a speech recognition algorithm used in this invention to extract dialogue content.

[0061] NFNet: NFNet is a deep learning neural network architecture used in this invention to extract visual features from videos.

[0062] word2vec: word2vec is a technique that maps words to a continuous vector space. In this invention, it is used to convert conversational text into multiple vectors.

[0063] librosa: The librosa library is a third-party Python library for speech signal processing, which is used in this embodiment of the invention to extract audio features;

[0064] Bi-GRU: Bi-GRU is a GRU network that performs bidirectional computation on the input sequence. In this embodiment of the invention, it is used for time series information aggregation.

[0065] The resampler module refers to the resampler module, which in this embodiment of the invention is used to resample the features of long sequence vectors.

[0066] Reference Figure 1 , Figure 2 and Figure 3 The multimodal sentiment analysis method based on large model video description in this invention may include a data preprocessing stage, a single-modal embedding stage, a cross-modal learning stage, and a modality fusion stage, and may specifically include the following steps S100 to S400.

[0067] S100. Obtain the video segment with sound and decompose it into video modal data, audio modal data and text modal data.

[0068] Step S100 may include the following steps S110 to S140.

[0069] S110. Extract several keyframes from the video clip with sound.

[0070] S120. Use the video frame corresponding to the key frame as video modal data.

[0071] S130. Use the audio corresponding to the keyframe as audio modal data.

[0072] S140. Perform video content detection and speech recognition on video frames and audio modal data to obtain text modal data.

[0073] For example, in the data preprocessing stage, the initial input is a video clip with sound. First, FFmpeg is used to extract T keyframes from the video at a frame rate of 5 frames per second. The audio associated with these T keyframes is used as audio modal data A. The video frames corresponding to each keyframe are used as video modal data V. Then, a video perception module composed of pre-trained models such as YOLOV8, FaceNet, and Whisper is used to perform video content detection and speech recognition on the keyframes to obtain a more detailed video description text as text modal data T.

[0074] S200. Perform single-modal embedding processing on the video modal data, audio modal data, and text modal data to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors.

[0075] In the single-modal embedding stage, since the data of different modalities have significant differences, different embedding methods must be selected. The embedding method can be represented as follows:

[0076]

[0077] Among them, X c The modal feature vectors of each modality are represented; emb() represents a single modal embedding operation; F6() represents an embedding operation using NFNet; word2vec() represents an embedding operation using word2vec; librosa() represents an embedding operation using the librosa library; c represents modal data; v represents video modal data; t represents text modal data; and a represents audio modal data.

[0078] Step S200 includes the following steps S210 to S240.

[0079] S210. Encode the video modal data from pixels to feature vectors to obtain the video sequence vector.

[0080] Video modal data is visually encoded from pixels to feature vectors using a pre-trained NFNet (F6). Each frame is processed by the F6 model to obtain two-dimensional features containing spatial information. The features of each frame are concatenated in the temporal dimension to output a three-dimensional tensor containing spatiotemporal information, which is then unfolded into a one-dimensional video sequence vector. Among them, L v Cv represents the total number of time steps in the video, and Cv represents the visual feature dimension. In other embodiments, other models capable of performing visual encoding from pixels to feature vectors can also be used for embedding processing.

[0081] S220. Generate a sequence of word vectors based on the text modal data.

[0082] The text modality data is used to generate word vector sequences using word2vec. Among them, L t C represents the number of words in the text. t This represents the dimension of text features. In other embodiments, other models capable of generating word vector sequences may also be used to generate word vector sequences.

[0083] S230. Extract audio features from the audio modal data to obtain an audio sequence vector.

[0084] The audio modal data was used to extract audio features such as Mel spectrum, spectral bandwidth, and zero-crossing rate using the librosa library. Among them, L a C represents the total number of time steps in the audio. a This represents the audio feature dimension. In other embodiments, other tools capable of extracting audio features may also be used for audio feature extraction.

[0085] S240. Perform temporal information aggregation processing on the video sequence vector, word vector sequence, and audio sequence vector to obtain video modal feature vector, text modal feature vector, and audio modal feature vector.

[0086] Specifically, step S240 may include the following steps S241 to S242.

[0087] S241. Calculate the cosine similarity between video sequence vectors, word vector sequences, and audio sequence vectors.

[0088] S242. Determine the feature vectors of several time steps with the highest information content based on cosine similarity to obtain video modal feature vectors, text modal feature vectors and audio modal feature vectors.

[0089] For steps S241 to S242, more specifically, in this embodiment of the invention, Bi-GRU is used for preliminary time-series information aggregation:

[0090]

[0091] Where, X′ c Let X represent the modality feature vector. Bi-GRU() indicates the temporal information aggregation operation using Bi-GRU. c represents the modality type. Three Bi-GRUs are needed to process the features X corresponding to the three modalities: audio modality a, text modality t, and visual modality v. a X t and X v Perform time series information aggregation separately to obtain X′ a 、X′ t and X′ v C represents the feature dimension.

[0092] Because long sequence vector features contain a lot of redundant information, the resampler module is used to extract it. First, the cosine similarity of the feature vectors is calculated to select the feature vectors from the time steps with the highest information content. The calculation formula is as follows:

[0093]

[0094] Here, similarity() is the operation for calculating cosine similarity; A represents one of the two vectors whose similarity needs to be calculated; B represents the other of the two vectors whose similarity needs to be calculated.

[0095] Specifically, considering that the text modality contains more semantic information than the visual and audio modalities, the vector selection for the visual modality needs to incorporate the feature vector from the last time step of the text modality. Then calculate all x with all time step vectors of the visual modality vi , i∈[1,L v ]:

[0096]

[0097] Then select the 30 time steps with the highest similarity:

[0098]

[0099] Similarly, the selection of audio modal vectors also references... Calculate the similarity between it and the feature vectors of all time steps, and extract the 30 time steps with the highest similarity:

[0100]

[0101] The selection of vectors for the text modality needs to comprehensively consider the information between the visual and audio modalities. Since there is redundancy or complementarity in the information between the visual and audio modalities, an adaptive approach is used for adjustment. The specific steps are as follows: First, initialize two learnable scalars a and b. Then, scale the two scalars using the tanh function. The tanh function takes values ​​in the range [-1, 1], which can both enhance information and suppress noise. Finally, the two scalars are correlated with the feature vectors of the visual and audio modalities, respectively. and Multiply the vectors, then sum the output vectors. Calculate the similarity between the vector sum and the text modality vectors, and extract 30 vectors based on the similarity score.

[0102]

[0103] i∈[1,Lt ], xs v ∈R 30×c

[0104] The final output is the text modal feature vector xs t Audio modal feature vector xs a and video modal feature vector xs v .

[0105] S300. Based on the video modal feature vector and the audio modal feature vector, perform information enhancement processing on the text modal feature vector to obtain the enhanced text modal feature vector.

[0106] Reference Figure 4 The cross-modal learning phase can be implemented in the query adapter module, where the text modality is augmented using vectors from the visual and audio modalities.

[0107] Step S300 includes the following steps S310 to S330.

[0108] S310. Determine the learnable scalar based on the feature vectors of each modality.

[0109] Considering that different modalities may conflict due to information differences, this embodiment of the invention resolves the modality conflict problem by assigning different priorities to different modalities. Specifically, a global convolution is used to compress the feature vectors of the three modalities into three learnable scalars α, β, and γ, and then normalization is performed using the softmax function. The calculation formula for the learnable scalars in this embodiment of the invention is as follows:

[0110] α = global-conv(xs) t )

[0111] β = global-conv(xs) a )

[0112] γ = global-conv(xs) v )

[0113] α,β,γ=softmax(α;β;γ)

[0114] Where α is a learnable scalar based on text modality feature vectors; β is a learnable scalar based on audio modality features; γ is a learnable scalar based on video modality feature vectors; global-conv() represents the global convolution operation; xs t Represents the text modal feature vector; xs a Represents the audio modal feature vector; xs vThis represents the video modal feature vector; softmax() represents the activation function.

[0115] By defining the three learnable scalars α, β, and γ, the three modes can be balanced during subsequent information value calculations.

[0116] S320. Configure the first parameter matrix and determine the first intermediate vector based on the modal feature vectors and the first parameter matrix.

[0117] Information enhancement in this embodiment of the invention is processed through an information enhancement layer. The information enhancement layer is constructed based on the traditional attention QKV mechanism. In this embodiment of the invention, two first parameter matrices, U and X, are added to this mechanism to construct an information enhancement layer with a structure of (q,k,v,u,x). u and x are used to introduce the influencing factors of the other two modalities besides the central modality. Here, the central modality is the text modality, and the other two are the audio modality and the visual modality, respectively.

[0118] Specifically, embodiments of the present invention will use audio modality xs a With the first parameter matrix W q Multiplying them yields the first intermediate vector Q, which in turn transforms the visual modality xs. v With the first parameter matrix W u Multiplying them yields the first intermediate vector U. The feature vectors of the text are then multiplied by the first parameter matrix W. k W v and W x Multiplying them yields the first intermediate vectors K, V, and X. The formulas for calculating the first intermediate vectors Q, K, V, U, and X are as follows:

[0119] Q = W q xs a

[0120] K = W k xs t

[0121] V = W v xs t

[0122] U = W u xs v

[0123] X = W x xs t

[0124] Among them, W q W k W u W v and W x All of these are first parameter matrices.

[0125] S330. Calculate the information value of each mode combination based on the first intermediate vector and the learnable scalar.

[0126] This invention employs a multi-type information combination design, selecting different modal combinations from different information combinations to achieve more diverse information interactions. Different information combinations can achieve the following:

[0127] info f1 =softmax(αQK) T +βXK T +γUK T V

[0128] info f2 =softmax(αQK) T +βXK T V

[0129] info f3 =softmax(βXK) T +γUK T V

[0130] Among them, info f1 This indicates the output of the first type of information combination; info f2 Indicates the output of the second type of information combination; info f3 Indicates the output of the third information combination; αQK T Let T represent the combination of the first intermediate vector Q and the first intermediate vector K, which are learnable scalars with α. T is the vector transpose, and other combinations follow the same pattern.

[0131] S340. Integrate the information values ​​to obtain the enhanced text modality feature vector.

[0132] Mining information hidden in text modalities using cross-modal queries:

[0133] xs′ t =Info(xs t ;xs a ;xs v α;β;γ)=f q,k,v,x,u (xs t ;xs a ;xs v α; β; γ)

[0134] Specifically, the first step is to calculate the output info of each information combination. fi Then, the average output is calculated to obtain the enhanced text modality feature vector, which is calculated using the following formula:

[0135]

[0136] Where, xs′ t Represents the enhanced text modal feature vector; info represents the information value; info fi Information values ​​representing the combinations of various modes.

[0137] S400 performs cross-modal fusion processing on the video modal feature vector, enhanced text modal feature vector, and audio modal feature vector in the large model block to obtain the sentiment analysis output value.

[0138] Reference Figure 5 In step S400, the feature vectors of each feature vector in this embodiment of the invention are fused into a large model block. A large model refers to a machine learning model with a large number of parameters and a complex structure. This embodiment of the invention constructs a large model block for modal fusion. Specifically, step S400 includes the following steps S410 to S430.

[0139] S410. Map the video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector into a common projection space.

[0140] Before performing modality fusion, a pre-trained cross-modal autoencoder (cross-AE) is used to map multiple modalities into a common projection space. The cross-AE consists of a cross-encoder and a cross-decoder, both of which are composed of multiple stacked linear layers. The cross-encoder projects the feature vectors of the text, audio, and visual modalities into a multimodal latent feature space, respectively. The cross-decoder then reconstructs the initial input vectors from the three modalities. It's important to note that the cross-decoder is only used during pre-training, and its regular output is the same as the cross-encoder's output, i.e., xs''. t xs' a and xs' v .

[0141] This invention employs an Adaptive Query layer for modality fusion. Adaptive Query, also based on an attention mechanism, shares a similar workflow; however, this invention improves upon it to obtain Adaptive Query. The biggest difference between Adaptive Query and the attention mechanism lies in splitting Q into {Q1, Q2, Q3} to process different modalities. Furthermore, {Q1, Q2, Q3} are entirely initialized and optimized using a backpropagation algorithm. Additionally, parameter matrices A, B, C, and D are added to obtain information about different modalities.

[0142] S420. Configure the second parameter matrix, and combine the second parameter matrix to perform cross-modal fusion vectors on the mapped modal feature vectors.

[0143] This step is initialization. First, the model needs to initialize three parameter matrices Q1, Q2, and Q3. Then, the inputs of the three modes (xs'') are... t xs' a xs' v ) respectively with W k W v W a W b W c W d Multiplying them together yields the second intermediate vector K, the second intermediate vector V, the second intermediate vector A, the second intermediate vector B, the second intermediate vector C, and the second intermediate vector D.

[0144] S430. Perform linear and nonlinear transformations on the cross-modal fusion vector to obtain the sentiment analysis output value.

[0145] This step involves calculating the output value, following the formula:

[0146] f=Adaptive Q(Q1;Q2;Q3;K;V;A;B;C;D)=softmax(Q1K T V+Q2A T B+Q3C T D)

[0147] Here, Adaptive Q represents an adaptive query, which adaptively generates query vectors to mine sentiment information in modalities. The output of Adaptive Q is a fusion vector f∈R. 30×C Then, linear and nonlinear transformations are performed through a multilayer perceptron (MLP) consisting of fully connected layers and activation functions. Finally, the output of the LM block (large model block) is f'. i , where i indicates the output of which LM block.

[0148] In some embodiments, the multimodal sentiment analysis method based on large model video description of the present invention may further include step S500.

[0149] S500. Construct a multimodal sentiment analysis model, train the multimodal sentiment analysis model to obtain a target multimodal sentiment analysis model; wherein, the target multimodal sentiment analysis model is used to perform sentiment analysis on audio video clips.

[0150] The multimodal sentiment analysis model constructed in this embodiment of the invention may include a model input part, a model processing part, and a model output part. The backbone of the multimodal sentiment analysis model is composed of alternating stacks of LM blocks and query adapters, wherein the input of the first query adapter is (xs t xs a xs v The input to the first LM block is the output of the first queryadapter (xs'). t xs a and XS v Its output is f'1, and in the subsequent f' i Will replace xs' t As input to the query adapter, the input to the query adapter at the nth (n≠1) level is (f' n-1 xs a xs v ).

[0151] The functions of each part of the model are further described below:

[0152] Model input: The initial source of the model is a video with sound. After a series of processing tools, video data, audio data, and text data are extracted. The video data provides visual information, the audio data provides audio information, and the text data provides text information containing some visual and audio information.

[0153] Model processing: The processing of each modality in the model consists of a data preprocessing stage, a single modality embedding stage, a modality fusion stage, and a context learning stage. The vision encoder, word embedding, and librosa are used for embedding the three modalities, respectively. Bi-GRU represents the temporal modeling process, resampler represents the temporal selection stage, and query adapter and LM block represent the modality fusion process.

[0154] Model output section: The model's output Y = (y 1,y 2 )∈R 1×2 These correspond to the probabilities of positive and negative emotions of the characters in the video.

[0155] After the model is trained, the output f of the last LM block is taken. out ∈R 30×c To make predictions.

[0156] Specifically, firstly, regarding f out Global average pooling is performed over the time dimension, and then the final predicted probability vector Y∈R is obtained by passing it through a feedforward layer combined with a softmax function. 1×2 .

[0157] Y=softmax(FFN(glob-avgpool(f out )))

[0158] The multimodal sentiment analysis model in this invention is a fully differentiable model, and the overall execution flow can be referred to... Figure 2 As shown, the algorithm can be optimized using backpropagation, and the cross-entropy function is used for downstream task training:

[0159]

[0160] in For the true emotional polarity situation, y i To predict the state of emotional polarity, by reducing The value can be used to complete the training of the model.

[0161] It should be noted that the parameters of the cross-AE module are frozen during the training of downstream tasks. Cross-AE needs to be trained separately and optimized using reconstruction loss. The reconstruction loss function is as follows:

[0162]

[0163] Furthermore, the cross-AE module performs self-supervised learning, allowing for the addition of unlabeled data for training.

[0164] On the other hand, such as Figure 6 As shown, this embodiment of the invention provides a multimodal sentiment analysis system based on large-model video description, including:

[0165] The first module is used to acquire video clips with sound and decompose them into video modal data, audio modal data and text modal data.

[0166] The second module is used to perform single-modal embedding processing on video modal data, audio modal data, and text modal data to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors.

[0167] The third module is used to perform information enhancement processing on the text modal feature vector based on the video modal feature vector and the audio modal feature vector to obtain the enhanced text modal feature vector;

[0168] The fourth module is used to perform cross-modal fusion processing on video modal feature vectors, enhanced text modal feature vectors, and audio modal feature vectors in the large model block to obtain sentiment analysis output values.

[0169] It should be noted that, in some embodiments, the system may further include the following modules:

[0170] The fifth module is used to construct a multimodal sentiment analysis model, train the multimodal sentiment analysis model to obtain a target multimodal sentiment analysis model; the target multimodal sentiment analysis model is used to perform sentiment analysis on audio video clips.

[0171] On the other hand, such as Figure 7 As shown, this embodiment of the invention also provides an electronic device, including: a processor and a memory; the memory is used to store a program; the processor executes the program to implement the method described above.

[0172] On the other hand, embodiments of the present invention also provide a computer storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the above-described method.

[0173] The embodiments of the present invention have the following beneficial effects:

[0174] 1. It can combine video and audio information from the original multimodal data to perform sentiment analysis, which contains more information such as the tone of voice and task status. The analysis results are more accurate and more in line with actual application scenarios.

[0175] 2. In the process of information enhancement and cross-modal fusion, two first intermediate vectors, U and X, are added to the traditional attention QKV mechanism, which can process the relevant data of the three modalities at the same time and achieve information enhancement. Furthermore, a learnable scalar is introduced for balancing, which helps to solve the modal conflict that may exist due to information differences between different modalities, so that the embodiments of the present invention can be better implemented.

[0176] The following is an application example of an embodiment of the present invention:

[0177] The process involves acquiring audio video clips and decomposing them into video modal data, audio modal data, and text modal data. Single-modal embedding is then performed on these data to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors. Information enhancement processing is then applied to the text modal feature vectors based on the video and audio modal feature vectors to obtain enhanced text modal feature vectors. Finally, cross-modal fusion processing is performed on the video modal feature vectors, enhanced text modal feature vectors, and audio modal feature vectors within a large model block to obtain the sentiment analysis output value.

[0178] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0179] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0181] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0182] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0183] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0184] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A multimodal sentiment analysis method based on large-model video description, characterized in that, include: Acquire a video segment with audio, and decompose the video segment with audio into video modal data, audio modal data, and text modal data; The video modal data, the audio modal data, and the text modal data are subjected to single-modal embedding processing to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors; Information enhancement processing is performed on the text modality feature vector based on the video modality feature vector and the audio modality feature vector to obtain the enhanced text modality feature vector; The video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector are subjected to cross-modal fusion processing in a large model block to obtain the sentiment analysis output value; The step of performing information enhancement processing on the text modal feature vector based on the video modal feature vector and the audio modal feature vector to obtain an enhanced text modal feature vector includes: A learnable scalar is determined based on the feature vectors of each modality; wherein, the learnable scalar is obtained by compressing the feature vectors of each model using a global convolution; Configure the first parameter matrix, and determine the first intermediate vector based on the feature vectors of each modality and the first parameter matrix; Calculate the information value of each mode combination based on the first intermediate vector and the learnable scalar; Information values ​​are integrated to obtain an enhanced text modality feature vector.

2. The multimodal sentiment analysis method based on large-model video description according to claim 1, characterized in that, The step of acquiring the audio video segment and decomposing the audio video segment into video modal data, audio modal data, and text modal data includes: Extract several keyframes from the video clip with sound; The video frames corresponding to the keyframes are used as video modal data; The audio corresponding to the keyframe is used as audio modal data; Video content detection and speech recognition are performed on the video frames and audio modal data to obtain text modal data.

3. The multimodal sentiment analysis method based on large-model video description according to claim 1, characterized in that, The step of performing single-modal embedding processing on the video modal data, the audio modal data, and the text modal data to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors includes: The video modal data is encoded from pixels to feature vectors to obtain a video sequence vector; Generate a word vector sequence based on the text modality data; Audio features are extracted from the audio modal data to obtain an audio sequence vector; The video sequence vector, the word vector sequence, and the audio sequence vector are subjected to temporal information aggregation processing to obtain video modal feature vector, text modal feature vector, and audio modal feature vector.

4. The multimodal sentiment analysis method based on large-model video description according to claim 3, characterized in that, The step of performing temporal information aggregation processing on the video sequence vector, the word vector sequence, and the audio sequence vector to obtain video modal feature vectors, text modal feature vectors, and audio modal feature vectors includes: Calculate the cosine similarity between the video sequence vector, the word vector sequence, and the audio sequence vector; Based on the cosine similarity, the feature vectors of several time steps with the highest information content are determined, resulting in video modal feature vectors, text modal feature vectors, and audio modal feature vectors.

5. The multimodal sentiment analysis method based on large-model video description according to claim 1, characterized in that, The process of performing cross-modal fusion processing on the video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector in a large model block to obtain sentiment analysis output values ​​includes: The video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector are mapped into a common projection space; Configure the second parameter matrix, and combine the second parameter matrix to perform cross-modal fusion vectors on the mapped modal feature vectors; The cross-modal fusion vector is subjected to linear and nonlinear transformations to obtain the sentiment analysis output value.

6. The multimodal sentiment analysis method based on large-model video description according to claim 1, characterized in that, The method further includes: A multimodal sentiment analysis model is constructed, and the multimodal sentiment analysis model is trained to obtain a target multimodal sentiment analysis model; wherein, the target multimodal sentiment analysis model is used to perform sentiment analysis on the audio video segment.

7. A multimodal sentiment analysis system based on large-model video description, characterized in that, include: The first module is used to acquire audio video segments and decompose the audio video segments into video modal data, audio modal data and text modal data; The second module is used to perform single-modal embedding processing on the video modal data, the audio modal data, and the text modal data to obtain video modal feature vectors, audio modal feature vectors, and text modal feature vectors; The third module is used to perform information enhancement processing on the text modal feature vector based on the video modal feature vector and the audio modal feature vector to obtain an enhanced text modal feature vector; The fourth module is used to perform cross-modal fusion processing on the video modal feature vector, the enhanced text modal feature vector, and the audio modal feature vector in the large model block to obtain the sentiment analysis output value; The third module is specifically used for: A learnable scalar is determined based on the feature vectors of each modality; wherein, the learnable scalar is obtained by compressing the feature vectors of each model using a global convolution; Configure the first parameter matrix, and determine the first intermediate vector based on the feature vectors of each modality and the first parameter matrix; Calculate the information value of each mode combination based on the first intermediate vector and the learnable scalar; Information values ​​are integrated to obtain an enhanced text modality feature vector.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 6.

9. A computer storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1 to 6.

Images