Text association type short video multi-modal sentiment recognition method and system

Abstract

The application discloses a text association type short video multi-modal emotion recognition method and system. The method is as follows: a video is divided into a plurality of candidate frame videos, and corresponding text is divided, and text and audio are extracted; a BERT model and a HuBERT model are used to train and learn features of word level, sentence level and inter-sentence relationship by using a large amount of unlabeled data; a cross attention mechanism and a self attention mechanism are used to filter a small amount of information and focus on important information; a non-attention Transformer model is used to learn auxiliary level information of a single mode; output features of an attention mechanism fusion module and a pre-feedback module are sent into a gated fusion network to fuse high and low layer semantic features, and a result of an emotion type is obtained. The system is composed of a video segmentation module, a video segment feature extraction module, an attention mechanism fusion module, a pre-feedback module, a gated fusion network module and an emotion classification module. The application solves the problem of feature loss in extraction and fusion, and improves the video emotion recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer vision video emotion recognition technology, and in particular to a text-related short video multimodal emotion recognition method and system. Background Technology

[0002] Emotion is a comprehensive expression of human behavior, thoughts, and states, and its components include at least three factors: emotional experience, emotional expression, and emotional physiology. From the perspective of expression, emotion recognition is mainly divided into two categories. The first category uses external emotional behaviors such as facial expressions, gestures, postures, and voice for recognition; while the second category uses internal physiological signal changes such as heart rate, respiration, blood pressure, body temperature, and electroencephalogram (EEG) signals for recognition.

[0003] From a computer science perspective, sentiment recognition is essentially a form of pattern recognition. It involves collecting data, building appropriate models to extract features from the collected information, and finally using these features to construct a classifier to obtain the sentiment recognition result. This process involves numerous technologies from various fields, such as signal processing, machine learning, and computer vision. Many methods and techniques, such as Bayesian networks, Gaussian mixture models, and hidden Markov models, are used for sentiment inference. Existing sentiment recognition methods can be broadly classified into three categories: knowledge-based methods, statistical methods, and hybrid methods.

[0004] Text sentiment analysis methods mainly include: dictionary-based sentiment analysis, machine learning-based sentiment analysis, and deep learning-based sentiment analysis. Dictionary-based sentiment analysis involves matching extracted sentiment words with an existing sentiment dictionary to calculate a sentiment polarity score, from which the sentiment polarity is derived. However, the existing sentiment dictionary needs continuous expansion to meet the demands. To improve the accuracy of sentiment classification and overcome the shortcomings of dictionary-based sentiment analysis, researchers began to study machine learning-based methods. This method uses a large number of training samples, employs machine learning algorithms for feature extraction, and uses a classifier for text sentiment classification. Commonly used classification models include SVM and Naive Bayes. Machine learning-based sentiment analysis methods often fail to fully utilize the semantic information of the context text when analyzing the sentiment of text content. Therefore, many researchers have studied deep learning-based sentiment analysis methods, with RNNs being widely used. Recurrent Neural Networks (RNNs) are powerful tools for processing sequences of various lengths (e.g., sentences, videos, and audio). Therefore, they are widely used in time series modeling. However, they assume that recent words are more meaningful than previous words, which may reduce the efficiency of capturing the semantics of the entire document. LSTM models overcome the limitations of RNNs. Gated Recurrent Unit (GRU) networks are also widely used due to their superior performance in capturing long-term dependencies. Furthermore, bidirectional versions such as BRNNs and Bi-LSTMs or BGRUs are also widely used for semantic capture. Additionally, attention mechanisms are introduced to improve classification accuracy. After using network structures such as CNNs, RNNs, and LSTMs for vector representation or feature extraction, attention mechanisms assign weights to the obtained features, highlighting important information and thus increasing classification accuracy. With the advent of BERT, more and more research is focusing on pre-trained models. Models trained on pre-trained datasets can achieve better sentiment analysis results through fine-tuning.

[0005] Speech-based sentiment analysis refers to extracting emotional features from valid speech signals and then classifying these features according to their emotional content. The general process of speech feature-based sentiment analysis includes data preprocessing, feature extraction, and speech sentiment classification. Data processing includes speech denoising, data framing, and windowing. Speech denoising involves cleaning speech data, removing noise, reducing the impact of the external environment on speech feature extraction, and lowering the signal-to-noise ratio. Common methods for speech denoising include using multiple filters to reduce noise generation. Due to the short-term stationarity of speech signals, they must be segmented into small segments for analysis. Data framing involves truncating the speech signal according to certain standards to form fixed-length speech data segments. After framing, there will be discontinuities at the beginning and end of each frame; therefore, the more frames there are, the greater the error compared to the original signal. Windowing addresses this problem by making the framed signal continuous, and ensuring that each frame exhibits the characteristics of a periodic function. Common window functions include rectangular windows, Hamming windows, and Hanning windows. Speech feature extraction refers to compressing speech data, that is, eliminating information that is irrelevant to the analysis and maximizing the content that contributes significantly to speech difference detection.

[0006] Numerous studies have shown that features characterizing speech emotion mainly include prosodic features, timbre features, and spectral features. Prosodic features include fundamental frequency, duration, and energy, while timbre features include spectroscopic and timbre-related characteristics. Spectral features mainly include parameters such as Mel-frequency cepstral coefficients (MFCC), linear predictive cepstral coefficients (LPCC), and logarithmic frequency power coefficients (LFPC). Currently, several toolkits have been developed to extract handcrafted features. For example, a commonly used tool is openSMILE, which can be used to extract acoustic features. In recent years, with the continuous development of deep learning, it has also been widely used in speech emotion analysis.

[0007] In the speech feature classification stage, traditional sentiment analysis algorithms mainly use machine learning algorithms. In recent years, deep learning techniques such as RNN, CNN, and LSTM have significantly improved the overall efficiency of speech sentiment recognition. However, none of the above studies have considered the specific temporal correlation between speech and text in discourse, that is, the discourse frame sequence and the word sequence are temporally consistent. Summary of the Invention

[0008] To address the issue that existing technologies fail to consider the specific temporal correlation between speech and text in discourse, this invention aims to propose a text-related multimodal emotion recognition method and system for short videos. This method integrates the emotional information of the text while utilizing the temporal features of the speech sequence in each frame of the video. Furthermore, the network not only fully considers temporal alignment but also pays attention to auxiliary information within the video, achieving emotion recognition and assessment of emotional richness.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0010] A text-related multimodal emotion recognition method for short videos includes the following steps:

[0011] Step S1: Divide the video with annotated text into several candidate frame videos according to the set frame rate, and divide the corresponding text into a complete semantic sequence. Extract the text and audio from the video according to the sequence.

[0012] Step S2: Use the pre-trained BERT model to extract video segment features; specifically, use the BERT-base-uncased model to extract text sequence features from the text sequence in the video segment of a set length, and at the same time use the pre-trained HuBERT model to extract the audio sequence features from the corresponding audio in sequence.

[0013] Step S3: The video segment features extracted in step S2 are fused with the text sequence features and audio sequence features using a cross-attention mechanism at the feature layer.

[0014] Step S4: Input the video segment features fused in Step S3 into the self-attention mechanism for global association weights, and then perform a weighted sum of the inputs to filter out a small amount of important information from a large amount of information and focus on this important information, while ignoring most of the unimportant information.

[0015] Step S5: Using the Transformer model without attention mechanism, extract auxiliary information of text and audio from the video segment features extracted in step S2.

[0016] Step S6: The information extracted in Step S4 and Step S5 is fed into the final gated fusion network as the output result, then concatenated, and a softmax classifier is used to obtain the probability matrix of the sentiment category to which the video belongs. After mapping it from the classification probability interval to the sentiment score interval, it is used as the sentiment type of the video part.

[0017] Furthermore, the cross-attention mechanism in step S3 includes the following steps:

[0018] Step S31: Obtain the speech feature representation X from the pre-trained HuBERT model. s =[s1,...,s k ]∈R T ×ds And a text feature representation X obtained from the pre-trained BERT model. t =[t1,...,t k ]∈R T×dtWe calculate the attention weights between the first audio vector and the first text vector, as shown in the formula below. Multiple attention heads in a multi-head attention mechanism allow the model to jointly focus on information from different representation subspaces at different locations.

[0019] Q s =X s W Q ,K t =X t W K V t =X t W V

[0020]

[0021] W Q ∈R ds×dk W K ∈R dt×dk W V ∈R dt×dv The weights are assigned to language and visual features. CroA is an operator for cross-modal attention using modular audio and text features, meaning they are temporally consistent with text features, thus enhancing audio features with complementary text information. Simultaneously, to strengthen the target modality's language features, residual connections are used to supplement audio modality information Q with text modality information. s The enhanced audio model characteristics are represented as follows:

[0022]

[0023] The results are then fed back into the self-attention layer, as shown in the formula below. In this layer, all keys and values ​​come from the output of the previous layer in the encoder. Each position in the encoder pays attention to all positions in the previous layer, thereby strengthening the aligned and integrated information.

[0024]

[0025]

[0026]

[0027] After standardization:

[0028]

[0029]

[0030] The above formula uses the interaction between different modules as a forward propagation, then feeds it into the network and normalizes it. Text is used as an auxiliary mode, where Xs is the time-aligned output of the audio target modulus transform; the feedback network includes two affine transformations, where ReLU is the activation function.

[0031] Step S32: Given the text representation Xt=[t1,...,t k ]∈R T×dt The audio representation is Xs = [s1,...,s] k ]∈R T ×ds Then, the alignment information between the first audio vector and the first text vector is calculated. The output Xt∈R is calculated using audio as the auxiliary pattern and text as the target pattern. T×dt Simultaneously, cross-modal alignment of each pair of time information is used to simulate the interaction between different modes.

[0032] Furthermore, step S5 employs a Transformer model without attention mechanism to extract auxiliary information from text and audio, specifically including the following steps:

[0033] Step S51, Given input X s =[s1,...,s k ]∈R T×ds The computational steps of the audio non-attention mechanism Transformer are similar to those of the attention mechanism. First, the input features are linearly transformed into:

[0034] Q s =X s W Q ,K t =X t W K V t =X t W V

[0035] Then perform the following operations:

[0036]

[0037] Here, Θ is the product of elements; σ q The activation function is specified, with the sigmoid function being the default; w∈R T×T For the learned pairwise feature positional biases; for each target position i, the attention-free Transformer computes its weighted average and combines the result with the element-wise product; the weights consist of key values ​​and a set of learned pairwise positional biases, which does not require computed and stored matrices, while maintaining global interaction between key values;

[0038] Step S52: Given the feature vector Xt=[t1,...,t of the input text. k ]∈R T×dt The result of the text processed by the attention-free Transformer mechanism is:

[0039]

[0040] Furthermore, step S6, which involves feeding the output result into the final gated fusion network, includes the following steps:

[0041] Step S61: To obtain efficient and accurate context information, a multimodal gate control module is proposed. First, standardized computation is used in the model; simple standardization achieves high-efficiency computation. Second, based on the standardized computation, gate control is used to adapt to context information. To prevent the gradient vanishing problem during training when the sigmoid function value is close to 0 or 1, it is introduced into gate adaptively, using 1+tanh(x) for activation, making the training process more stable.

[0042] Step S62: Use x = cancat(X) S ,X T )+cancat(Y S ,Y T First, X S With X T Y S With Y T Perform vector concatenation, then send the concatenated result x into the multimodal gate control module. The next step is as follows:

[0043]

[0044] For the concatenated bimodal features, the L2 transform is calculated. The L2 model can aggregate global contextual information in each feature. ε is a very small constant to avoid the problem of differentiation at zero.

[0045] Step S63: When resources are limited and training performance is stable, a normalization method is then used to calculate the competition relationship between neurons. The normalization operation is as follows:

[0046]

[0047] scalars are used To increase the value of the ratio in the above formula, and to prevent the ratio from becoming too small when C is too large.

[0048] At the same time, the gating module adopts adaptive characteristics of the original features:

[0049]

[0050] Step S64: Classify video sentiment using the classification probability interval; the method is as follows: given the obtained representation We first apply a fully connected layer with corrected linear units (ReLU) for the nonlinear transformation g, and then use a softmax output layer to obtain... Sentiment classification for discourse.

[0051]

[0052]

[0053] Here, W g W e yes The weights of the nonlinear (i.e., tanh) transformation of g;

[0054] Step S65, the sentiment weight w and the validity threshold t are determined based on the global loss function. The parameters are learned using gradient descent, where To predict sentiment category, y i , where N represents the actual emotion category and N is the total number of samples.

[0055] Furthermore, a text-related short video multimodal emotion recognition system includes:

[0056] The video segmentation module is used to segment a video with annotated text into several candidate frame images according to a set frame rate and extract the text and audio containing complete semantics.

[0057] The video segment feature extraction module is used to extract text sequence features of video segments of a set length using a pre-trained BERT model, namely the Bert-base-uncased model, and at the same time to extract audio sequence features of the corresponding video segments in sequence using a pre-trained HuBERT model.

[0058] The cross-attention mechanism module is used to fuse text sequence features and audio sequence features at the feature layer.

[0059] The self-attention mechanism module is used to perform global association weights, then perform a weighted sum of the inputs, filter out a small amount of important information from a large amount of information, and focus on this important information while ignoring most of the unimportant information;

[0060] The prefeedback module connects the unimodal output and the multimodal gating module. It acquires unimodal feature information through a Transformer model without attention mechanisms and sends it to the gating fusion layer to assist in sentiment type judgment. Its learned positional bias is limited to local regions while maintaining global connectivity. Therefore, the prefeedback network focuses on the global information of a single modality in the video.

[0061] The gating fusion module is used to further fuse the high-level semantic information extracted by the attention mechanism with the low-level sequence features extracted by the feature extraction module, so as to perform sentiment classification more accurately.

[0062] The emotion recognition module is used to classify emotions in videos; the emotion weights and thresholds are obtained by self-learning using gradient descent based on the loss function of the predicted emotion category and the true emotion category of the sample data.

[0063] Furthermore, the system includes at least one computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the text-related short video multimodal emotion recognition method.

[0064] Compared with the prior art, the present invention, by adopting the above technical solution, has the following beneficial effects:

[0065] 1. The model in this invention employs the Bert-base-uncased model and the HuBERT model, which can extract video text and audio information. It explicitly utilizes the comparison information between speech and text, and simultaneously obtains the temporal alignment between the two modalities, thereby making the information fusion more complementary. Then, a pre-feedback network is used to perform feature layer fusion and higher-order fusion on the features, as well as a gated fusion layer, to automatically learn the contribution of each representation of the speech and text modes.

[0066] 2. The temporal alignment in this invention can provide a more discriminative representation for sentiment classification, the prefeedback network enables the network to focus on auxiliary information of each modality, and the gated fusion layer can effectively fuse multiple representations in an interpretable manner.

[0067] 3. This invention has significant robustness in judging emotion categories and scoring the richness of emotions in video emotion recognition, and improves the accuracy of video emotion recognition to a certain extent. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of a text-related short video multimodal emotion recognition method and system according to the present invention;

[0069] Figure 2 This is a schematic diagram of the gated fusion layer structure provided in an embodiment of the present invention;

[0070] Figure 3 This is a training flowchart of an embodiment of the present invention. Detailed Implementation

[0071] The present invention is described below based on embodiments, but the invention is not limited to these embodiments. In the detailed description of the invention below, certain specific details are described in detail. Those skilled in the art will fully understand the invention even without these details. To avoid obscuring the essence of the invention, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0072] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings:

[0073] like Figure 1 As shown in the figure, this invention discloses a text-related short video multimodal emotion recognition method and system. This method uses the BERT model to extract text sequence features from the video, and simultaneously uses the HuBERT model to extract the corresponding audio sequence features. A cross-attention mechanism is then used to fuse the two features, which are then fed into a self-attention mechanism to obtain high-level semantic features. A pre-feedback network is used to process the two input modalities. Finally, the output of the pre-feedback network and the output of the self-attention mechanism are fed into a final gated fusion network to further fuse the high-level semantic features, and finally, the video is classified for emotion. The parameters of the HuBERT and BERT network models involved are pre-trained and will not be elaborated upon in this invention. The video emotion recognition process of this embodiment is described in detail below, specifically including the following steps:

[0074] Step A involves dividing the video containing annotated text into several candidate frame videos at a set frame rate, such as 25 frames per second, and extracting the text and audio containing complete semantics.

[0075] Step B involves inputting the segmented text sequence from Step A into the BERT model [Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805,2018.] to extract temporal features from the text of the short video. Simultaneously, the HuBERT network model is used to sequentially extract semantic features from each audio sequence. The features extracted by the BERT model are used as text sequence features, and the features of the corresponding text extracted by HuBERT are used as audio sequence features. The two types of features are then fused. Specifically, this includes:

[0076] Step B1: Each consecutive L frames of the video candidate frames extracted in Step A are considered a sequence. If there are fewer than L frames, the last frame is oversampled and padded to form a sequence. The BERT network is then used to extract features for each sequence. In feature extraction for each sequence, the input sequence is marked with [CLS] and [SEP] tags at the beginning and end of the text by a word segmenter, respectively. The returned input is a dictionary. The BERT output includes the hidden state of the last layer of the model and the last hidden layer of the output sequence, i.e., the [CLS] label. The shape of the hidden state of the last layer of the model represents (number of training set samples, sequence length, hidden capacity). The last hidden layer of the output sequence is obtained by fully connected operations on the features corresponding to the [CLS] tag in the hidden state of the last layer of the model, which can generally be considered to summarize the information of the entire sentence. The output is a text sequence feature containing video temporal information with a dimension of 768.

[0077] Step B2: Input the audio from each sequence into Hubert in order to calculate the features of the audio corresponding to each sequence, and output audio sequence features with a dimension of 768 that contain video temporal information.

[0078] Step C involves fusing the 768-dimensional text sequence features from Step B1 with the 768-dimensional audio sequence features from Step B2 as the underlying input features representing the video segment. The input feature fusion layer performs the fusion. Specifically, this includes:

[0079] Step C1 reduces the feature dimension to 30 and cross-combines them before feeding them into an attention mechanism network [Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need [C] / / arXiv.arXiv, 2017.] for feature fusion. While characterizing the emotion of each sequence in the video, each element of the text sequence features and audio sequence features input is considered with different weight parameters, thus paying more attention to the parts similar to the input elements and suppressing other useless information. This strengthens the understanding of the video content at the feature level, and fully considers the temporal alignment of text and audio from the same video segment, using their complementary information to improve the accuracy of emotion recognition.

[0080] Step C2 involves inputting the video segment feature representation fused in Step C1 into a self-attention mechanism [Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need [C] / / arXiv.arXiv, 2017.] for further feature extraction and fusion. The video segment feature representation fused in Step C1, i.e., the text and audio, and the audio and text components, are cross-combined and then fed into the self-attention mechanism modules. Each self-attention mechanism module receives n inputs and returns n outputs. The self-attention mechanism allows each input to interact with each other and then finds the inputs that should receive more attention. The output of the self-attention module is the aggregation of these interactions and the attention score.

[0081] Step D involves fusing the 768-dimensional text sequence features from Step B1 with the 768-dimensional audio sequence features from Step B2 as the underlying input features representing the video segment. This fusion is then fed into a pre-feedback network for global attention, yielding the corresponding global information. Specifically, this includes:

[0082] In step D1, for each target position i, the attention-free Transformer computes its weighted average and combines the result with a query using element-wise multiplication. Specifically, the learned weights consist of a key and a set of acquired pairwise positional biases. This eliminates the need to compute and store computationally complex matrices, while the learned positional biases are confined to local regions and global connectivity is maintained. Therefore, the prefeedback network focuses on the global information of a single modality in the video, outputting the global information of that single modality.

[0083] Step E involves fusing the features from the feature fusion layer completed in step C, and then combining the output of step D with the output of the gated fusion network for high-level semantic fusion. Specifically, this includes:

[0084] Step E1, for the speech modality, yields two representations: X S As a speech representation processed by the self-attention mechanism, Y S This serves as the last hidden state in the pre-feedback model. For textual modality, there are two other similar representations X. T With Y T To utilize these representations, a novel gated fusion layer is used to automatically learn the contribution of each representation. The governing equations of the gated fusion layer are as follows:

[0085] x = cancat(X) S ,X T )+cancat(Y S ,Y T )

[0086] First X SWith X T Y S With Y T Perform vector concatenation, then send the concatenated result x into the multimodal gate control module. The next step is as follows:

[0087]

[0088] For the concatenated bimodal features, the L2 transform is calculated. The L2 model can aggregate global contextual information in each feature. ε is a very small constant to avoid the problem of differentiation at zero.

[0089] When resources are limited and training performance is stable, a normalization method is then used to calculate the competition between neurons. The normalization operation is as follows:

[0090]

[0091] scalars are used To increase the value of the ratio in the above formula, and to prevent the ratio from becoming too small when C is too large.

[0092] At the same time, the gating module adopts adaptive characteristics of the original features:

[0093]

[0094] The result of the above formula is used as input for the final representation of the subsequent sentiment classification layer.

[0095] Step E2: Use a softmax classifier to obtain the probability matrix of the sentiment category to which each video belongs, map it to the classification probability interval [0,1], and normalize it to ensure that the sum is 1, which serves as the sentiment classification of the video part.

[0096] This invention discloses a text-related short video multimodal emotion recognition system, comprising: a video segmentation module for segmenting a video with annotated text into several candidate frame images according to a set frame rate and extracting text and audio sequentially; a video segment feature extraction module for extracting text sequence features from the video using a BERT model and audio sequence features from the sequence using a HuBERT model; a time alignment module for fusing the two features using a cross-attention mechanism and then feeding them into a self-attention mechanism to obtain high-level semantic features; and a pre-feedback module for processing the two input modalities to obtain their globally associated modal auxiliary information. Finally, the output of the pre-feedback module and the output of the self-attention mechanism are fed into a final gated fusion network to further fuse high- and low-level semantic features, and finally perform emotion classification on the video.

[0097] The above-described embodiment of a text-related short video multimodal emotion recognition system can be used to execute the above-described embodiment of a text-related short video emotion recognition method. The technical principles, the technical problems solved, and the technical effects produced are similar. The specific working process and related descriptions of the above-described text-related short video multimodal emotion recognition can be referred to the corresponding process in the above-described embodiment of a text-related short video multimodal emotion recognition method, and will not be repeated here.

[0098] Those skilled in the art will understand that the modules in the embodiments can be adaptively modified and placed in one or more systems different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components.

[0099] Based on the same inventive concept, a text-related short video multimodal emotion recognition system includes at least one computer device. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned text-related short video multimodal emotion recognition method.

[0100] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A text-related multimodal emotion recognition method for short videos, characterized in that, Includes the following steps: Step S1: Divide the video with annotated text into several candidate frame videos according to the set frame rate, and divide the corresponding text into a complete semantic sequence. Extract the text and audio from the video according to the sequence. Step S2: Use the pre-trained BERT model to extract video segment features; use the Bert-base-uncased model to extract text sequence features from the text sequence in the video segment of a set length, and at the same time use the pre-trained HuBERT model to extract the audio sequence features from the corresponding audio in sequence. Step S3: The video segment features extracted in step S2 are fused with the text sequence features and audio sequence features using a cross-attention mechanism at the feature layer. Step S4: Input the video segment features fused in Step S3 into the self-attention mechanism for global association weights, then perform a weighted sum of the inputs, filter out important information, focus on these important information, and ignore unimportant information; Step S5: Using the Transformer model without attention mechanism, extract auxiliary information of text and audio from the video segment features extracted in step S2. Step S6: The information extracted in Steps S4 and S5 is fed into the final gated fusion network as the output result and spliced ​​together. The softmax classifier is used to obtain the probability matrix of the emotion category to which the video belongs. After mapping it from the classification probability interval to the emotion score interval, it is used as the emotion type of the video part.

2. The text-related short video multimodal emotion recognition method according to claim 1, characterized in that, The cross-attention mechanism employed in step S3 includes the following steps: Step S31: Speech feature representation obtained from the pre-trained HuBERT model =[ , ..., ]∈ And a text feature representation X obtained from the pre-trained BERT model. t =[ , ..., ]∈ The attention weights between the first audio vector and the first text vector are calculated as shown in the following formula. Multiple attention heads in the cross-attention mechanism allow the model to jointly focus on information from different representation subspaces at different locations. ∈ , ∈ , ∈ The weights are assigned to language and visual features; CroA is an operator for cross-modal attention using modular audio and text features. Simultaneously, residual connections are used to supplement audio modal information with text modal information. The enhanced audio model characteristics are represented as follows: The results are then fed back into the self-attention layer, as shown in the formula below. In this layer, all keys and values ​​come from the output of the previous layer in the encoder. Each position in the encoder pays attention to all positions in the previous layer, thereby strengthening the aligned and integrated information. After standardization: The above formula takes the interaction between different modules as the forward propagation, then feeds it into the network and normalizes it, with text as the auxiliary mode, and Xs as the time-aligned output of the audio target modulus transformation. The network includes two affine transformations, where ReLU is the activation function. Step S32, given text representation Xt=[ , ..., ]∈ Audio representation Xs=[ , ..., ]∈ Then, the alignment information between the first audio vector and the first text vector is calculated, and the output Xt∈ is calculated using audio as the auxiliary mode and text as the target mode. Meanwhile, cross-modal alignment of each pair of time information is used to simulate the interaction between different modes.

3. The text-related short video multimodal emotion recognition method according to claim 2, characterized in that, Step S5 employs a Transformer model without attention mechanisms to extract auxiliary information from text and audio, including the following steps: Step S51, Given input X s =[ , ..., ] ∈ The computation steps of the audio attention-free mechanism Transformer are as follows: First, the input features are linearly transformed into: Then perform the following operations: Here It is the product of elements; This is the activation function; the default is the sigmoid function. For the learned pairwise feature positional biases; for each target position i, the attention-free Transformer computes its weighted average and combines the result with the element-wise product; the weights consist of key values ​​and a set of learned pairwise positional biases, without the need to compute and store matrices, while maintaining global interaction between key values; Step S52: Given the feature vector Xt of the input text, [ , ..., ]∈ The result of the text processed by the attention-free Transformer mechanism is: 。 4. The text-related short video multimodal emotion recognition method according to claim 1, characterized in that, Step S6, which involves feeding the output into the final gated fusion network, includes the following steps: Step S61: Use standardized computation in the model. Simple standardization is used to perform the computation. Based on the standardized computation, gating is used to adapt to the context information. In order to prevent the gradient vanishing problem from easily occurring during training when the sigmoid function value is close to 0 or 1, gating adaptation is introduced, and 1+tanh(x) is used for activation. Step S62, using First and , and Perform vector concatenation, then send the concatenated result x into the multimodal gate control module. The next step is as follows: For the concatenated bimodal features, the L2 transform is calculated. The L2 model can aggregate global contextual information in each feature, and ε is a very small constant. Step S63: Calculate the competition relationship between neurons using a normalization method. The normalization operation is as follows: scalars are used To increase the value of the ratio in the above formula, and to prevent the ratio from becoming too small when C is too large. At the same time, the gating module adopts adaptive characteristics of the original features: Step S64: Classify video sentiment using a method that assigns classification probability intervals; given the obtained representation First, a fully connected layer with corrected linear units (ReLU) is applied for nonlinear transformation. And use a softmax output layer to obtain Sentiment classification for discourse: , yes and The weights of the nonlinear transformation; Step S65: Based on the global loss function The sentiment weights are learned using gradient descent. and validity threshold ,in To predict sentiment categories, , where N represents the actual sentiment category and N is the total number of samples.

5. A text-related short video multimodal emotion recognition system, characterized in that, The system includes: The video segmentation module is used to segment a video with annotated text into several candidate frame images according to a set frame rate and extract the text and audio containing complete semantics. The video segment feature extraction module is used to extract text sequence features from video segments of a set length using a pre-trained BERT model, namely the Bert-base-uncased model, and at the same time, to extract audio sequence features from the corresponding video segments in sequence using a pre-trained HuBERT model. The cross-attention mechanism module is used to fuse text sequence features and audio sequence features at the feature layer, focusing on the temporal correlation between text features and audio features; The self-attention mechanism module is used to perform global correlation weights, then perform a weighted sum of the inputs, filter important information, and focus on this important information while ignoring most unimportant information; The prefeedback module connects the single-modal output and the multimodal gating module. It obtains single-modal feature information through a Transformer model without attention mechanism and sends it to the gating fusion layer to assist in emotion type judgment. Its learned positional bias is limited to local regions and maintains global connectivity. The prefeedback network focuses on the global information of a single modality in the video. The gated fusion module is used to further fuse the high-level semantic information extracted by the attention mechanism with the low-level sequence features extracted by the feature extraction module, and obtain the contribution of each feature to the classification to perform sentiment classification. The emotion recognition module is used to classify emotions in videos; the emotion weights and thresholds are obtained by self-learning using gradient descent based on the loss function of the predicted emotion category and the true emotion category of the sample data.

6. The text-related short video multimodal emotion recognition system according to claim 5, characterized in that, The system includes at least one computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the text-related short video multimodal emotion recognition method.

Images