A multimodal humor recognition method

Abstract

The present invention provides a humor recognition method for multimodality, which relates to the technical field of multimodal humor recognition. In one implementation, the present invention treats the introduced external knowledge as a separate modality and fuses it with text, audio, and video modalities. First, the dimensions are aligned through a fully connected layer, and then the splicing is input into the Transformer to learn the interactive relationship between the four modalities. At the same time, the modal fusion is optimized with the help of a cross-attention mechanism, thereby solving the problems of external knowledge introduction and modal fusion schemes. In another implementation, data enhancement is performed through a graph attention network to make up for the current scarcity of data sets. The related modalities are fused and spliced, and then sent to the cross-attention network, and then further fused through a multi-head self-attention layer. Finally, it is input into the maximum pooling layer to obtain a single feature representation of the fusion vector, and the predicted probability distribution of humor polarity is obtained, thereby obtaining a more effective modal fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal humor recognition, and in particular to a multimodal humor recognition method. Background Art

[0002] Early research on humor was primarily limited to textual data, employing machine learning methods for humor recognition, constructing feature vectors and training classifiers to identify humor in text. With the rise of neural networks, many traditional machine learning methods have been gradually abandoned, replaced by deep learning architectures such as the convolutional neural networks employed by Chen and Lee, and the Transformer architecture employed by Weller and Seppi. In the 2021 Humorous and Offensive Text Identification and Evaluation Task held by the Association for Computational Linguistics, Xie et al. used the decoupled-attention decoding-enhanced BERT model, DeBERTa, to predict humor tags. Samuel Akrah used the pretrained RoBERTa model to complete the task. Bo Huang et al. used a model combining ALBERT, CNN, and the Tf-Idf algorithm for humor recognition.

[0003] With the increasing availability of video and audio content, researchers are increasingly focusing on how to effectively fuse information from different modalities (such as text, speech, and images) to improve the accuracy and robustness of humor recognition. The bimodal modal fusion network proposed by Wei Han et al. is an end-to-end model that effectively fuses and separates data from different modalities. This approach helps to process the correlations and differences between modalities in multimodal data, thereby achieving more accurate humor recognition.

[0004] In addition, the Transformer model based on humor-rich knowledge published in AAAI is one of the important advances in this field. This study introduced external knowledge (such as ambiguity theory and superiority theory) into multimodal humor recognition for the first time, providing deeper background information for humor understanding, and greatly expanding the expressive power and recognition accuracy of existing models. The optimal transmission method has been applied in the research of Shraman Pramanick et al. to solve the problem of limited humor data. Through this method, the model can better handle humor data from different sources and different types, and improve recognition performance in data-scarce conditions. In 2023, Wu et al. proposed a modal fusion model based on the attention mechanism. First, the information of each modality is independently encoded, and then the interaction information between different modalities is combined through the attention mechanism. This method can dynamically focus on the most important parts of humor recognition in different modalities, thereby enhancing the performance of the model. In addition, the emotion-knowledge-enhanced self-supervised learning method proposed by Fan et al. introduces emotion knowledge and non-verbal behavior information into multimodal humor recognition. Through the training of the self-supervised learning model, it can effectively learn the emotional characteristics and non-verbal behavior cues of humor without relying on a large amount of labeled data, making the model more natural and close to real language usage scenarios when processing humor.

[0005] Although humor recognition has been explored extensively, the field of humor recognition based on multimodal data still faces challenges.

[0006] First, external knowledge can help the model understand context and identify humorous elements such as puns and metaphors, thereby improving recognition accuracy. However, current work lacks the introduction of external knowledge. Second, the commonly used methods for multimodal fusion currently mostly use weighted addition, splicing, or averaging, which can easily cause information redundancy or loss, thereby affecting experimental results. Previous fusion schemes also did not consider the relationship between different modalities. Finally, the two currently available benchmark datasets, MUStARD and UR-FUNNY, are both relatively small in size, which will result in the experiment being unable to accurately capture important features or information in the data during training, and may even cause overfitting and poor generalization. Summary of the Invention

[0007] In view of the shortcomings of the prior art, the present invention aims to propose a multimodal humor recognition method, comprising:

[0008] Step 1: Acquire multiple input video data, wherein the input video data includes input text, face image and audio information;

[0009] Step 2: For each input video data in step 1, perform word segmentation and vector embedding on the input text to obtain the final text vector, and input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t ;

[0010] Step 3: For each input video data in step 1, align and calibrate the face image using affine transformation technology to obtain an aligned face image; crop the face part of the aligned face image to obtain a first image, input the first image into a deep convolutional neural network (CNN) to obtain a face feature vector, perform L2 normalization on the face feature vector to obtain a normalized face feature vector, and select action units and face shape parameter features from the normalized face feature vector as the final face image feature f v ;

[0011] Step 4: For the audio information in step 1, extract the pitch information from the audio information, extract the pitch information through COVAREP, obtain multiple pitch features, and process the multiple pitch features through SPS software to obtain the audio feature f after dimensionality reduction. a ;

[0012] Step 5: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and then extract the humorous features and text modal features U t , the final face image feature f v and audio features f a , get the visual modality feature U v , audio modal features U a and humor modality characteristics U f ;

[0013] Step 6: Text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Processing is performed to obtain a predicted probability distribution P representing humor polarity, which is specifically achieved through step A or step B:

[0014] Step A: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Input the linear layer to obtain the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ aand the final humor modal feature U′ f , for the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f The spliced ​​results are processed through a multi-head self-attention mechanism and a fully connected layer to obtain a predicted probability distribution P representing the humor polarity of the input video data;

[0015] Step B: Attention network is used to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Update the updated features and pass them through a cross-attention mechanism, a linear transformation, a multi-head cross-attention mechanism, and a multi-head self-attention layer to obtain a predicted probability distribution P that represents the humor polarity of the input video data;

[0016] Step 7: Use the predicted probability distribution P as the predicted value and obtain the true value representing the humor polarity corresponding to the input video data. Calculate the loss function value based on the predicted value and the true value. This is achieved using the following formula:

[0017]

[0018] Among them, BinaryCrossEntropy is the loss function value, N is the number of input video data, y i is the true value of the i-th input video data, y′ i is the predicted value of the i-th input video data;

[0019] The embedding vector matrix, deep convolutional neural network (CNN), Transformer encoder, linear layer parameters W1 and b1, and fully connected layer parameters W2 and b2 are updated through the loss function value until the loss function value is less than the preset threshold. At this time, the output predicted probability distribution P is the humor polarity prediction result of the input video data.

[0020] Optionally, step 2 specifically includes:

[0021] Step 2.1: For each input video data in step 1, the input text is segmented by the WordPiece word segmenter of the ALBERT model to obtain the segmented text S′={(t1, t2, ..., t j ),(p1,p2,...,p k )}, where (t1,t2,...,t j) is the context of the input text, t1 represents the first word of the context, t2 represents the second word of the context, t j represents the jth word in the context, j represents the total number of subwords in the context, (p1, p2, ..., p k ) is the punchline of the input text, p1 represents the first word of the punchline, p2 represents the second word of the punchline, and p k The kth word representing the punchline, where k is the total number of subwords of the punchline, is marked on the segmented text. Specifically, a [CLS] tag is added at the beginning of the segmented text, a [SEP] tag is added between the context and the punchline, and a [SEP] tag is added at the end of the segmented text. The resulting text sequence X is [CLS]t1...t j [SEP]p1...p k [SEP];

[0022] Step 2.2: Convert the text sequence into the form of embedded vectors to obtain the embedding vector matrix X1=E [CLS] E1...E j E [SEP] E′1...E′ k E [SEP] , where E [CLS] is the embedding vector of [CLS], E1 is the embedding vector of t1, E j t j The embedding vector, E [SEP] is the embedding vector of [SEP], E′1 is the embedding vector of p1, and E′ k For p k The embedding vector, E [SEP] is the embedding vector of [SEP]; for the context and punchline in the embedding vector matrix, different segment embeddings are added to obtain the first embedding representation. Position embedding is added to each word in the first embedding representation to obtain the final text vector embedding representation. The position embedding represents the position of the word in the first vector embedding representation.

[0023] Step 2.3: Input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t .

[0024] Optionally, step 5 specifically includes:

[0025] Step 5.1: For each input video data in step 1, extract the contextual humor features and the humor features of the punchline from the input text, and convert the contextual humor features and the humor features of the punchline into a form to obtain the converted humor features X f =[PAD]C f [SEP]Pf , where C f is the humor feature corresponding to the context, P f is the humor feature corresponding to the punchline, [PAD] is a placeholder, and [SEP] is used to convert C f With P f separate;

[0026] Step 5.2: Use the P2FA forced alignment model to extract the time corresponding to each word in the text segmented in step 2.1, and then perform face image feature f according to the time corresponding to each word. v and audio features f a Segmentation is performed and the segmented face image features are represented as X v =[PAD]C v [SEP]P v , C v is the facial image feature corresponding to the context, P v is the facial image feature corresponding to the laugh point, and the segmented audio feature is represented as X a =[PAD]C a [SEP]P a , C a is the audio feature corresponding to the context, P a The audio features corresponding to the funny points;

[0027] Step 5.3: The segmented face image features X v Input Transformer encoder to obtain visual modality feature U v , specifically expressed by the following formula:

[0028] U v =TransformerEncoder(X v );

[0029] The segmented audio feature X a Input Transformer encoder to obtain audio modality feature U a , specifically expressed by the following formula:

[0030] U a =TransformerEncoder(X a );

[0031] Set humor feature X f Input Transformer encoder to get humor modality feature U f , specifically expressed by the following formula:

[0032] U f=TransformerEncoder(X f ).

[0033] Optionally, step A specifically includes:

[0034] Step A1: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f , respectively input into the linear layer to obtain the final text modality feature U′ of the same dimension t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f , the formula of the linear layer is expressed as:

[0035] Y=XW1+b1;

[0036] Among them, W1 is the learning weight matrix and b1 is the bias vector;

[0037] Specifically, the text modality feature U t , visual modality features U v , audio modal features U a and humor modal features into X to obtain the final text modal feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f ;

[0038] Step A2: Final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f Splice and get the matrix M=[U′ t ,U′ v ,U′ a ,U′ f ], through the multi-head self-attention mechanism, the matrix M is processed to obtain M′=[U″ t ,U″ v ,U″ a ,U″ f ], which is specifically achieved through the following formula:

[0039] M=MultiHead(M,θ)=Concat(head1,...,head h )Wo ;

[0040] θ={W q ,W k ,W v ,W o );

[0041] Among them, MultiHead represents multi-head splicing, head1 represents the first attention head, head h represents the hth attention head, W q ,W k ,W v ,W o is the attention mechanism parameter, where each attention head is expressed as:

[0042]

[0043] M=K=V=Q;

[0044] Among them, head c represents the cth attention head, Attention represents the self-attention mechanism, is the attention mechanism parameter;

[0045] Replace U″ in M′ t ,U″ v ,U″ a ,U″ f Combine and get the joint vector U final , specifically expressed by the following formula:

[0046]

[0047] The joint vector U final The input is fed into the fully connected layer and the predicted probability distribution P representing the humor polarity is obtained through the Softmax function. It is specifically expressed by the following formula:

[0048] P=SoftMax(U final W2+b2);

[0049] Among them, W2 and b2 are the parameters of the fully connected layer.

[0050] Optionally, step B specifically includes:

[0051] Step B1: Use graph attention network to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Update to get the updated text modal feature f t, updated humor modal features f f , updated visual modality features f v and the updated audio modality features f a ;

[0052] Step B2: Through the cross attention mechanism, the updated text modality feature f t and the updated humor modal feature f f Perform modal fusion to obtain the first fusion feature f t,f , specifically expressed by the following formula:

[0053]

[0054] Through the cross attention mechanism, the updated visual modality feature f v and the updated audio modality features f a Perform modal fusion to obtain the second fusion feature f a,v , specifically expressed by the following formula:

[0055]

[0056] Step B3: respectively perform the first fusion feature f t,f and the second fusion feature f a,v Perform linear transformation to obtain the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is implemented by the following formula:

[0057]

[0058]

[0059] in, and is the weight matrix, K1 is the first fusion feature f t,f The key feature matrix, V1 is the first fusion feature f t,f The feature matrix of the value of K2 is the second fusion feature f a,v The key feature matrix, V2 is the second fusion feature f a,v The characteristic matrix of the values ​​of ;

[0060] Step B4: Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, and the outputs of all attention heads are aggregated to obtain the first output O1 and the second output O2;

[0061] Step B5: Combine O1 with the first fusion feature f t,f Connect and get the updated modal feature f′ t,f , O2 is combined with the second fusion feature f a,v Connect and get the updated modal feature f′ a,v , specifically expressed by the following formula:

[0062] f′ t,f =O1+f t,f ;

[0063] f′ a,v =O2+f a,v ;

[0064] Step B6: f′ t,f and f′ a,v Input into the multi-head self-attention layer in sequence to obtain the processed information feature f a,v,f,t , input the processed information features into the maximum pooling layer to obtain the final feature f final , the final feature f final Input the softmax layer to obtain the predicted probability distribution P that represents the humor polarity of the input video data, which is specifically expressed by the following formula:

[0065] P=softmax(f final W5+b5);

[0066] Among them, W5 and b5 are the parameters of the softmax layer.

[0067] Optionally, step B1 specifically includes:

[0068] The text modality features U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Perform feature extraction and dependency analysis to obtain a node feature matrix for text mode, a node feature matrix for visual mode, a node feature moment for audio mode, a node feature matrix for humor mode, and an adjacency matrix. The adjacency matrix represents the grammatical dependency relationship between words.

[0069] The node feature matrix of the text modality includes a text feature vector corresponding to each word, and each text feature vector is used as a node of a text graph to obtain an initial text graph; the node feature matrix of the visual modality includes a visual feature vector corresponding to each word, and each visual feature vector is used as a node of a visual graph to obtain an initial visual graph; the node feature matrix of the audio modality includes an audio feature vector corresponding to each word, and each audio feature vector is used as a node of an audio graph to obtain an initial audio graph; the node feature matrix of the humor modality includes a humor feature vector corresponding to each word, and each humor feature vector is used as a node of a humor graph to obtain an initial humor graph;

[0070] The weight W3 of the linear transformation is initialized through the uniform distribution formula. For each node in the initial text graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix. The initialized linear transformation weight W3 is multiplied with the text feature vector of the node and the text feature vector of an adjacent node respectively. The result of the multiplication with the text feature vector of the node and the result of the multiplication with the text feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes. This is specifically achieved through the following formula:

[0071] e mn =a([W3A m ||W3A n ]),n∈N m ;

[0072] Among them, e mn is the correlation coefficient between node m and node n, || represents the connection operation, a(·) represents the mapping operation, A m is the text feature vector corresponding to node m, A n is the text feature vector corresponding to node n, N m Represents the subscript set of nodes connected to node m;

[0073] Normalize the correlation coefficient between the node and its adjacent nodes to obtain the normalized attention coefficient, which is specifically achieved through the following formula:

[0074]

[0075] Among them, α mn is the normalized attention coefficient of node m and node n, e mr is the correlation coefficient between node m and node r;

[0076] The normalized attention coefficients are weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated text modality feature f t , which is specifically achieved through the following formula:

[0077]

[0078] Among them, A′ m (l) represents the output vector of node m in the lth attention head, that is, the updated text modality feature f t , L represents the total number of attention heads, σ(·) represents the activation function, represents the normalized attention coefficient of the lth attention head, represents the weight matrix of the lth attention head;

[0079] Similarly, for each node in the initial visual graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the visual feature vector of the node and the visual feature vector of an adjacent node are multiplied by the initialized linear transformation weight W3 respectively, and the result of multiplying the visual feature vector of the node and the result of multiplying the visual feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated visual modal feature f v ;

[0080] For each node in the initial audio graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the audio feature vector of the node and the audio feature vector of an adjacent node, and the result of multiplication with the audio feature vector of the node and the result of multiplication with the audio feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node, and the correlation coefficient between the node and the adjacent node is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated audio modal feature f a ;

[0081] For each node in the initial humor graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the humor feature vector of the node and the humor feature vector of an adjacent node, and the result of multiplying with the humor feature vector of the node and the result of multiplying with the humor feature vector of the adjacent node is spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes. The correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated humor modal feature f f .

[0082] Optionally, step B4 specifically includes:

[0083] Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, which is implemented by the following formula:

[0084]

[0085] Among them, d k Represents the dimension of the key, Attention h (Q1, K2, V2) is the output of the hth attention head after multi-head attention calculation of Q1, K2 and V2. h (Q2, K1, V1) is the output of the hth attention head after multi-head attention calculation of Q2, K1 and V1; H represents the total number of attention heads, is the index of the h-th attention head in Q1; is the index of the h-th attention head in Q2; is the index of the h-th attention head in K2; for The index of the h-th attention head in ; is the index of the h-th attention head in V1; is the index of the h-th attention head in V2; for The transpose of for The transpose of

[0086] Aggregate the outputs of all attention heads to get, which is achieved by the following formula:

[0087] O1=Concat([Attention 1 (Q1,K2,V2),...,Attention H (Q1,K2,V2)]);

[0088] O2=Concat([Attention 1 (Q2,K1,V1),...,Attention H (Q2,K1,V1)]);

[0089] Among them, H represents the total number of attention heads, O1 is the output of all attention heads obtained by multi-head attention calculation of Q1 with K2 and V2, and O2 is the output of all attention heads obtained by multi-head attention calculation with K1 and V1.

[0090] The beneficial effects of adopting the above technical solution are:

[0091] In one implementation, the present invention treats the introduced external knowledge as a separate modality and fuses it with text, audio, and video modalities. First, the dimensions of the different modal features are aligned through a fully connected layer, and then they are spliced ​​and input into the Transformer to learn the interactive relationship between the four modalities and output a single feature representation. The present invention optimizes modal fusion with the help of a cross-attention mechanism, solving the problems of external knowledge introduction and modal fusion solutions. In another implementation, the present invention uses a graph attention network to perform data enhancement to make up for the scarcity of current datasets. Furthermore, the relationship between the four modalities is considered, and it is believed that the humor-related modality is closely related to the text modality, and the audio and visual modalities are closely related, so the related modalities are fused and spliced. Then, it is sent to the cross-attention network to learn the complementary representation of the features and output them, and then further fused through the multi-head self-attention layer. Finally, it is input into the maximum pooling layer to obtain a single feature representation of the fusion vector, thereby obtaining a more effective modal fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 1 is a flow chart of a multimodal humor recognition method according to an embodiment of the present invention;

[0093] Figure 2 Schematic diagram of a process for obtaining a predicted probability distribution P in an embodiment of the present invention;

[0094] Figure 3 Schematic diagram of another process for obtaining a predicted probability distribution P in an embodiment of the present invention. DETAILED DESCRIPTION

[0095] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0096] To address the problems of the existing technology, the present invention provides a multimodal humor recognition method. To address the two issues of how to introduce external knowledge and how to integrate external knowledge, the present invention proposes a humor recognition method based on HTT. To introduce external knowledge into the humor recognition model, the present invention summarizes the currently recognized mainstream humor theories and uses them as the theoretical basis for acquiring humorous external knowledge. The present invention selects appropriate features for encoding and representation as the injection of external knowledge. To create an effective modality fusion solution, the present invention treats the introduced external knowledge as a separate modality and integrates it with text, audio, and video modalities.

[0097] Specifically, the present invention solves the problems of external knowledge introduction and modal fusion schemes by proposing a humor recognition method based on HTT. In order to introduce external knowledge into the humor recognition model, the present invention summarizes the currently recognized mainstream humor theories as the theoretical basis for external knowledge acquisition. Finally, there are three theories adopted: superiority theory from the perspective of social behavior, release theory from the perspective of psychological analysis, and incongruity theory from the perspective of psychological cognition. Taking the release theory as the starting point, combined with the data set provided by SemEval 2021Task 7 and the Word Art Creator word cloud tool, an offensive dictionary was constructed to calculate the offensive level of each word as a humor feature; taking the superiority theory as the starting point, the SentiWordNet sentiment dictionary was used to calculate the sentiment score of each word as a humor feature; finally, two humor-related features were constructed based on the incongruity theory. The first is to calculate ambiguity. Specifically, the number of semantic sets of each word is calculated through WordNet, and the top T words are selected as candidate ambiguous words. Then, the cosine similarity function is selected to measure the semantic distance, and the standard deviation of these similarity values ​​is calculated and the standard deviation is used as a measure of ambiguity. The second method is to use the NRC VAD sentiment dictionary to extract the sentiment polarity score of each word. The present invention encodes the four features mentioned above as external knowledge injection. In order to create an effective modality fusion solution, the present invention treats the introduced external knowledge as a separate modality and fuses it with text, audio, and video modalities. First, the dimensions of the different modal features are aligned through the fully connected layer, and then the splicing is input into the Transformer to learn the interaction relationship between the four modalities and output a single feature representation.

[0098] To address data shortages and modal fusion challenges, a humor recognition method based on GACA is proposed. This method utilizes a graph attention network for data augmentation, updating the data within the network to obtain enhanced feature data. This method optimizes modal fusion using a cross-attention mechanism, fully considering the relationships between the four modalities and effectively fusing and splicing related modalities.

[0099] Specifically, the present invention solves the problems of data shortage and modal fusion solutions by proposing a humor recognition method based on GACA. In order to make up for the problem of the scarcity of current data sets, the present invention proposes a solution for data enhancement using a graph attention network. Specifically, for the text modality, each word is regarded as a data point, and then an adjacency matrix is ​​constructed based on the dependency relationship between words. Then, the vector representation of the word is obtained by Glove embedding to obtain a point feature matrix. For the other three modalities, because they have been aligned with the text words in the data processing stage, the adjacency matrix and the text are consistent, and the point feature matrix is ​​obtained through the corresponding feature extraction tool. Finally, the two matrices are sent together to the graph attention network to complete the data update and obtain enhanced feature data. In order to create a more effective modal fusion solution, the present invention optimizes modal fusion with the help of a cross-attention mechanism. First, the relationship between the four modalities is considered. It is believed that the humor-related modality is closely related to the text modality, and the audio and visual modalities are closely related, so the related modalities are fused and spliced. It is then fed into the cross-attention network to learn the complementary representation of the features and output them, which are then further fused through the multi-head self-attention layer and finally input into the maximum pooling layer to obtain a single feature representation of the fused vector.

[0100] Based on the above scheme, the present invention provides a multimodal humor recognition method, combined with Figure 1 , specifically including the following steps:

[0101] Step 1: Acquire multiple input video data, wherein the input video data includes input text, face image and audio information;

[0102] Step 2: For each input video data in step 1, perform word segmentation and vector embedding on the input text to obtain the final text vector, and input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t ;

[0103] Step 2.1: For each input video data in step 1, the input text is segmented by the WordPiece word segmenter of the ALBERT model to obtain the segmented text S′={(t1, t2, ..., t j ),(p1,p2,...,p k)}, where (t1,t2,...,t j ) is the context of the input text, t1 represents the first word of the context, t2 represents the second word of the context, t j represents the jth word in the context, j represents the total number of subwords in the context, (p1, p2, ..., p k ) is the punchline of the input text, p1 represents the first word of the punchline, p2 represents the second word of the punchline, and p k The kth word representing the punchline, where k is the total number of subwords of the punchline, is marked on the segmented text. Specifically, a [CLS] tag is added at the beginning of the segmented text, and a [SEP] tag is added between the context and the punchline to separate the two information sources, the context and the punchline, so that modeling can be performed based on the context and the punchline. A [SEP] tag is added at the end of the segmented text to obtain the text sequence X' = [CLS]t1...t j [SEP]p1...p k [SEP];

[0104] Step 2.2: Convert the text sequence into the form of embedded vectors to obtain the embedding vector matrix X1=E [CLS] E1...E j E [SEP] E′1...E′ k E [SEP] , where E [CLS] is the embedding vector of [CLS], E1 is the embedding vector of t1, E j t j The embedding vector, E [SEP] is the embedding vector of [SEP], E′1 is the embedding vector of p1, and E′ k For p k The embedding vector, E [SEP] The embedding vector for [SEP] is obtained. Different segment embeddings are added to the context and punchline in the embedding vector matrix to obtain the first embedding representation. Adding different segment embeddings helps the model recognize and correctly process two different text inputs. Position embeddings are added to each word in the first embedding representation to obtain the final text vector embedding representation. Position embeddings represent the position of a word in the first vector embedding representation. Adding position embeddings helps the model learn contextual relationships and better understand sentence semantics.

[0105] Step 2.3: Input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t .

[0106] Specifically, the final text vector embedding representation is input into the ALBERT model, and the input is processed by the Encoder module in the multi-layer stacked Transformer structure. The Encoder module of each layer uses a multi-head self-attention mechanism to capture the global dependencies between different positions in the input sequence, and perform context modeling on the text semantic information, thereby obtaining high-quality text modal features U containing rich semantic information. t .

[0107] Step 3: For each input video data in step 1, since the face in the face image may be tilted, the face needs to be aligned and corrected before the image is input into the convolutional neural network. Therefore, the present invention aligns and calibrates the face image through affine transformation technology to obtain an aligned face image;

[0108] Specifically, facial key points are detected on the face image. These key points are usually located at the corners of the eyes, nose, and facial contours. Then, the key points and affine transformation technology are used to align and calibrate the face, adjusting the face to the correct position, thereby reducing errors caused by different postures.

[0109] The facial part of the aligned facial image is cropped to obtain a first image, and the first image is input into a deep convolutional neural network (CNN) to obtain a facial feature vector, wherein the deep convolutional neural network (CNN) is composed of a batch input layer and a deep convolutional neural network (CNN); the facial feature vector is L2 normalized to obtain a normalized facial feature vector, and the L2 normalization is used to map the feature into a hyperspherical Euclidean space to ensure that the feature vector has a unit length; in the normalized facial feature vector, the action unit and the face shape parameter features are selected as the final facial image feature f v ;

[0110] Step 4: For the audio information in step 1, extract pitch information from the audio information, extract the pitch information through COVAREP, and obtain multiple pitch features. Specifically, the pitch information can be extracted through the methods of spectrum envelope estimation, beam tracking, sinusoidal modeling, phase processing and glottal analysis in COVAREP. The multiple pitch features obtained include Melcepstral coefficient, fundamental frequency, voiced / unvoiced segment, normalized amplitude quotient, quasi-open quotient, glottal source parameter, harmonic model and phase distortion, formant and other features; since some features such as cepstrum features and glottal features show high correlation in some analysis aspects, the present invention processes multiple pitch features through SPS software to obtain the audio feature f after dimensionality reduction. a ;

[0111] Step 5: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and then extract the humorous features and text modal features U t , the final face image feature f v and audio features f a , get the visual modality feature U v , audio modal features U a and humor modality characteristics U f ;

[0112] Step 5.1: For each input video data in step 1, extract the contextual humor features and the humor features of the punchline from the input text, and convert the contextual humor features and the humor features of the punchline into a form to obtain the converted humor features X f =[PAD]C f [SEP]P f , where C f is the humor feature corresponding to the context, P f is the humor feature corresponding to the punchline, [PAD] is a placeholder, and [SEP] is used to convert C f With P f separate;

[0113] Step 5.2: Use the P2FA forced alignment model to extract the time corresponding to each word in the text segmented in step 2.1, and then perform face image feature f according to the time corresponding to each word. v and audio features f a Segmentation is performed and the segmented face image features are represented as X v =[PAD]C v [SEP]P v , C v is the facial image feature corresponding to the context, P v is the facial image feature corresponding to the laugh point, and the segmented audio feature is represented as X a =[PAD]C a [SEP]P a , C a is the audio feature corresponding to the context, P a The audio features corresponding to the funny points;

[0114] Among them, humor feature X f , segmented face image features X v and the segmented audio features X a It can be uniformly expressed as X E , where E = {a, v, f}, X E ∈R T×dE , d Eis the dimension of the corresponding feature, T represents the number of words;

[0115] Step 5.3: The segmented face image features X v Input Transformer encoder to obtain visual modality feature U v , specifically expressed by the following formula:

[0116] U v =TransformerEncoder(X v );

[0117] The segmented audio feature X a Input Transformer encoder to obtain audio modality feature U a , specifically expressed by the following formula:

[0118] U a =TransformerEncoder(X a );

[0119] Set humor feature X f Input Transformer encoder to get humor modality feature U f , specifically expressed by the following formula:

[0120] U f =TransformerEncoder(X f ).

[0121] Among them, the visual modality feature U v , audio modal features U a and humor modality characteristics U f It can be uniformly expressed as U E , d UE The output dimension of the Transformer encoder corresponding to the modality feature E is the dimension of the feature in the corresponding modality.

[0122] Step 6: Text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Processing is performed to obtain a predicted probability distribution P representing humor polarity, which is specifically achieved through step A or step B:

[0123] The present invention provides a humor recognition method based on HTT, which is specifically implemented through step A.

[0124] Step A: Transform the text modality feature U t , visual modality features Uv , audio modal features U a and humor modality characteristics U f Input the linear layer to obtain the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f , for the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f The spliced ​​results are processed through a multi-head self-attention mechanism and a fully connected layer to obtain a predicted probability distribution P representing the humor polarity of the input video data;

[0125] Combine Figure 2 , which is achieved through the following steps:

[0126] Step A1: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f , respectively input into the linear layer to obtain the final text modality feature U′ of the same dimension t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f , the formula of the linear layer is expressed as:

[0127] Y=XW1+b1;

[0128] Among them, W1 is the learning weight matrix and b1 is the bias vector;

[0129] Specifically, the text modality feature U t , visual modality features U v , audio modal features U a and humor modal features into X to obtain the final text modal feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f ;

[0130] Step A2: Final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′a and the final humor modal feature U′ f Splice and get the matrix M=[U′ t ,U′ v ,U′ a ,U′ f ], through the multi-head self-attention mechanism, the matrix M is processed to obtain M′=[U″ t ,U″ v ,U″ a ,U″ f ], which is specifically achieved through the following formula:

[0131] M′=MultiHead(M,θ)=Concat(head1,...,head h )W o ;

[0132] θ={W q ,W k ,W v ,W o );

[0133] Among them, MultiHead represents multi-head splicing, head1 represents the first attention head, head h represents the hth attention head, W q ,W k ,W v ,W o is the attention mechanism parameter, where each attention head is expressed as:

[0134]

[0135] M=K=V=Q;

[0136] Among them, head c represents the cth attention head, Attention represents the self-attention mechanism, is the attention mechanism parameter;

[0137] The multi-head self-attention mechanism used in this invention allows the features of each modality to perceive and focus on the features of other modalities. Through this self-attention mechanism, the features of each modality can extract relevant information from the other modalities, thereby achieving cross-modal interaction and obtaining fused multimodal features. This cross-modal information fusion has a synergistic effect on overall humor recognition, thereby improving the accuracy of humor recognition.

[0138] Replace U″ in M′ t ,U″ v ,U″ a ,U″ f Combine and get the joint vector Ufinal , specifically expressed by the following formula:

[0139]

[0140] The joint vector U final The input is fed into the fully connected layer and the predicted probability distribution P representing the humor polarity is obtained through the Softmax function. It is specifically expressed by the following formula:

[0141] P=SoftMax(U final W2+b2);

[0142] Among them, W2 and b2 are the parameters of the fully connected layer.

[0143] The predicted probability distribution P includes the probability of humor, the probability of non-humor, and the probability of neutrality.

[0144] The present invention also provides a GACA-based humor recognition method, which is specifically implemented through step B.

[0145] Step B: Attention network is used to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Update the updated features and pass them through the cross attention mechanism, linear transformation, multi-head cross attention mechanism and multi-head self-attention layer to obtain the predicted probability distribution P that characterizes the humor polarity of the input video data; Combined Figure 3 , specifically including the following steps:

[0146] Step B1: Use graph attention network to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Update to get the updated text modal feature f t , updated humor modal features f f , updated visual modality features f v and the updated audio modality features f a ;

[0147] Step B1 specifically includes:

[0148] The text modality features U t , visual modality features U v , audio modal features U a and humor modality characteristics U fPerform feature extraction and dependency analysis to obtain a node feature matrix for text mode, a node feature matrix for visual mode, a node feature moment for audio mode, a node feature matrix for humor mode, and an adjacency matrix. The adjacency matrix represents the grammatical dependency relationship between words.

[0149] Specifically, the node feature matrix obtains the features of each word through GloVe word embedding, and the adjacency matrix is ​​obtained by dependency analysis using SpyCy.

[0150] The node feature matrix of the text modality includes a text feature vector corresponding to each word, and each text feature vector is used as a node of a text graph to obtain an initial text graph; the node feature matrix of the visual modality includes a visual feature vector corresponding to each word, and each visual feature vector is used as a node of a visual graph to obtain an initial visual graph; the node feature matrix of the audio modality includes an audio feature vector corresponding to each word, and each audio feature vector is used as a node of an audio graph to obtain an initial audio graph; the node feature matrix of the humor modality includes a humor feature vector corresponding to each word, and each humor feature vector is used as a node of a humor graph to obtain an initial humor graph;

[0151] Initialize the weight W3 of the linear transformation through the uniform distribution formula. The uniform distribution formula is expressed as:

[0152]

[0153] Among them, n in is the number of input layer nodes, n out is the number of output layer nodes, gain is an optional scaling factor used to adjust the initialization standard deviation.

[0154] For each node in the initial text graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix. The initialized linear transformation weight W3 is multiplied with the text feature vector of the node and the text feature vector of an adjacent node respectively. The result of multiplication with the text feature vector of the node and the result of multiplication with the text feature vector of the adjacent node are concatenated. The concatenated result is mapped through a one-layer feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes. This is specifically achieved through the following formula:

[0155] e mn =a([W3A m ||W3A n ]),n∈N m ;

[0156] Among them, e mn is the correlation coefficient between node m and node n, || represents the connection operation, a(·) represents the mapping operation, Am is the text feature vector corresponding to node m, A n is the text feature vector corresponding to node n, N m Represents the subscript set of nodes connected to node m;

[0157] Normalize the correlation coefficient between the node and its adjacent nodes to obtain the normalized attention coefficient, which is specifically achieved through the following formula:

[0158]

[0159] Among them, α mn is the normalized attention coefficient of node m and node n, e mr is the correlation coefficient between node m and node r;

[0160] The normalized attention coefficients are weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated text modality feature f t , which is specifically achieved through the following formula:

[0161]

[0162] Among them, A′ m (l) represents the output vector of node m in the lth attention head, that is, the updated text modality feature f t , L represents the total number of attention heads, σ(·) represents the activation function, represents the normalized attention coefficient of the lth attention head, represents the weight matrix of the lth attention head;

[0163] Similarly, for each node in the initial visual graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the visual feature vector of the node and the visual feature vector of an adjacent node are multiplied by the initialized linear transformation weight W3 respectively, and the result of multiplying the visual feature vector of the node and the result of multiplying the visual feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated visual modal feature f v ;

[0164] For each node in the initial audio graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the audio feature vector of the node and the audio feature vector of an adjacent node, and the result of multiplication with the audio feature vector of the node and the result of multiplication with the audio feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node, and the correlation coefficient between the node and the adjacent node is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated audio modal feature f a ;

[0165] For each node in the initial humor graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the humor feature vector of the node and the humor feature vector of an adjacent node, and the result of multiplying with the humor feature vector of the node and the result of multiplying with the humor feature vector of the adjacent node is spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes. The correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated humor modal feature f f .

[0166] Step B2: Through the cross attention mechanism, the updated text modality feature f t and the updated humor modal feature f f Perform modal fusion to obtain the first fusion feature f t,f , specifically expressed by the following formula:

[0167]

[0168] in, d t is the output dimension of the encoder corresponding to the updated text modality feature, d f The updated humor modality feature corresponds to the encoder output dimension;

[0169] Through the cross attention mechanism, the updated visual modality feature f v and the updated audio modality features f a Perform modal fusion to obtain the second fusion feature f a,v , specifically expressed by the following formula:

[0170]

[0171] in d v is the output dimension of the encoder corresponding to the updated visual modality feature, d a The output dimension of the encoder corresponding to the updated audio modality feature;

[0172] Step B3: respectively perform the first fusion feature f t,f and the second fusion feature f a,v Perform linear transformation to obtain the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is implemented by the following formula:

[0173]

[0174] in, and is the weight matrix, which can be learned, and K1 is the first fusion feature f t,f The key feature matrix, V1 is the first fusion feature f t,f The feature matrix of the value of K2 is the second fusion feature f a,v The key feature matrix, V2 is the second fusion feature f a,v The characteristic matrix of the values ​​of ;

[0175] Step B4: Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, and the outputs of all attention heads are aggregated to obtain the first output O1 and the second output O2;

[0176] Wherein, step B4 specifically includes:

[0177] Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, which is implemented by the following formula:

[0178]

[0179] Among them, d k Represents the dimension of the key, Attention h(Q1, K2, V2) is the output of the hth attention head after multi-head attention calculation of Q1, K2 and V2. h (Q2, K1, V1) is the output of the hth attention head after multi-head attention calculation of Q2, K1 and V1; H represents the total number of attention heads, is the index of the h-th attention head in Q1; is the index of the h-th attention head in Q2; is the index of the h-th attention head in K2; for The index of the h-th attention head in ; is the index of the h-th attention head in V1; is the index of the h-th attention head in V2; for The transpose of for The transpose of

[0180] Aggregate the outputs of all attention heads to get, which is achieved by the following formula:

[0181] O1=Concat([Attention 1 (Q1,K2,V2),...,Attention H (Q1,K2,V2)]);

[0182] O2=Concat([Attention 1 (Q2,K1,V1),...,Attention H (Q2,K1,V1)]);

[0183] Among them, H represents the total number of attention heads, O1 is the output of all attention heads obtained by multi-head attention calculation of Q1 with K2 and V2, and O2 is the output of all attention heads obtained by multi-head attention calculation with K1 and V1.

[0184] Step B5: Combine O1 with the first fusion feature f t,f Connect and get the updated modal feature f′ t,f , O2 is combined with the second fusion feature f a,v Connect and get the updated modal feature f′ a,v , specifically expressed by the following formula:

[0185] f′ t,f =O1+f t,f ;

[0186] f′a,v =O2+f a,v ;

[0187] Step B6: f′ t,f and f′ a,v Input into the multi-head self-attention layer in sequence to obtain the processed information feature f a,v,f,t , the multi-head self-attention layer is used to update each modal feature of the input. A normalization and residual connection layer is added after each self-attention sub-layer in the multi-head self-attention layer; the processed information features are input into the maximum pooling layer to obtain the final feature f final , the final feature f final Input the softmax layer to obtain the predicted probability distribution P that represents the humor polarity of the input video data, which is specifically expressed by the following formula:

[0188] P=softmax(f final W5+b5);

[0189] Among them, W5 and b5 are the parameters of the softmax layer.

[0190] Step 7: Use the predicted probability distribution P as the predicted value and obtain the true value representing the humor polarity corresponding to the input video data. Calculate the loss function value based on the predicted value and the true value. This is achieved using the following formula:

[0191]

[0192] Among them, BinaryCrossEntropy is the loss function value, N is the number of input video data, y i is the true value of the i-th input video data, y′ i is the predicted value of the i-th input video data;

[0193] The loss function used in the present invention is binary cross entropy loss.

[0194] The embedding vector matrix, deep convolutional neural network (CNN), Transformer encoder, linear layer parameters W1 and b1, and fully connected layer parameters W2 and b2 are updated through the loss function value until the loss function value is less than the preset threshold. At this time, the output predicted probability distribution P is the humor polarity prediction result of the input video data.

[0195] Humor recognition from multimodal data aims to detect whether humor exists in given data. As a special task in the field of natural language processing, it is extremely interesting but also faces severe challenges.

[0196] Two key points in the present invention are as follows:

[0197] This paper proposes a humor recognition method based on HTT. This method summarizes currently accepted mainstream humor theories and uses them as a theoretical basis for acquiring external knowledge about humor. Appropriate features are selected for encoding and representation to serve as the injection of external knowledge. This method treats external knowledge as a separate modality and fuses it with text, audio, and video modalities. The key point of this invention is to address the issues of external knowledge introduction and modality fusion solutions.

[0198] This paper proposes a humor recognition method based on GACA. It uses a graph attention network for data augmentation to address the current scarcity of datasets. Furthermore, it optimizes modal fusion with a cross-attention mechanism to create a more efficient modal fusion solution.

[0199] Based on the above scheme, the present invention uses datasets with language, acoustic and visual modalities for experiments. Currently, the only open source datasets that meet the conditions are UR-FUNNY and MUStARD, which provide labeled data for natural language processing and machine learning model training, especially for identifying humorous contexts in videos or texts. In order to evaluate the performance of the model, the present invention evaluates it through binary classification accuracy and F1 score. Combined with Table 1, the comparative experiment shows that the HTT model achieves an accuracy improvement of 0.06% and 1.34% compared to the optimal baseline model. Combined with Table 2, the GACA model achieves an accuracy improvement of 0.77% and 1.98% compared to HTT, thereby verifying the effectiveness and advancement of the model of the present invention.

[0200] Table 1 Experimental results of HTT and baseline comparison

[0201]

[0202] Table 2 Comparison results of GACA and baselines

[0203]

[0204]

[0205] The above description is merely a preferred embodiment of the present disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multimodal humor recognition method, characterized in that: include: Step 1: Acquire multiple input video data, wherein the input video data includes input text, face image and audio information; Step 2: For each input video data in step 1, perform word segmentation and vector embedding on the input text to obtain the final text vector, and input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t ; Step 3: For each input video data in step 1, align and calibrate the face image using affine transformation technology to obtain an aligned face image; crop the face part of the aligned face image to obtain a first image, input the first image into a deep convolutional neural network (CNN) to obtain a face feature vector, perform L2 normalization on the face feature vector to obtain a normalized face feature vector, and select action units and face shape parameter features from the normalized face feature vector as the final face image feature f v ; Step 4: For the audio information in step 1, extract the pitch information from the audio information, extract the pitch information through COVAREP, obtain multiple pitch features, and process the multiple pitch features through SPS software to obtain the audio feature f after dimensionality reduction. a ; Step 5: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and then extract the humorous features and text modality features U t , the final face image feature f v and audio features f a , get the visual modality feature U v , audio modal features U a and humor modality characteristics U f ; Step 6: Text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Processing is performed to obtain a predicted probability distribution P representing humor polarity, which is specifically achieved through step A or step B: Step A: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Input the linear layer to obtain the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f , for the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f The spliced ​​results are processed through a multi-head self-attention mechanism and a fully connected layer to obtain a predicted probability distribution P representing the humor polarity of the input video data; Step B: Attention network is used to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Update the updated features and pass them through a cross-attention mechanism, a linear transformation, a multi-head cross-attention mechanism, and a multi-head self-attention layer to obtain a predicted probability distribution P that represents the humor polarity of the input video data; Step 7: Use the predicted probability distribution P as the predicted value and obtain the true value representing the humor polarity corresponding to the input video data. Calculate the loss function value based on the predicted value and the true value. This is achieved using the following formula: Among them, BinaryCrossEntropy is the loss function value, N is the number of input video data, y i is the true value of the i-th input video data, y′ i is the predicted value of the i-th input video data; The embedding vector matrix, deep convolutional neural network (CNN), Transformer encoder, linear layer parameters W1 and b1, and fully connected layer parameters W2 and b2 are updated through the loss function value until the loss function value is less than the preset threshold. At this time, the output predicted probability distribution P is the humor polarity prediction result of the input video data.

2. A multimodal humor recognition method according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: For each input video data in step 1, the input text is segmented by the WordPiece word segmenter of the ALBERT model to obtain the segmented text S′={(t1, t2, ..., t j ),(p1,p2,...,p k )}, where (t1,t2,...,t j ) is the context of the input text, t1 represents the first word of the context, t2 represents the second word of the context, t j represents the jth word in the context, j represents the total number of subwords in the context, (p1, p2, ..., p k ) is the punchline of the input text, p1 represents the first word of the punchline, p2 represents the second word of the punchline, and p k The kth word representing the punchline, where k is the total number of subwords of the punchline, is marked on the segmented text. Specifically, a [CLS] tag is added at the beginning of the segmented text, a [SEP] tag is added between the context and the punchline, and a [SEP] tag is added at the end of the segmented text. The resulting text sequence X is [CLS]t1...t j [SEP]p1...p k [SEP]; Step 2.2: Convert the text sequence into the form of embedded vectors to obtain the embedding vector matrix X1=E [CLS] E1...E j E [SEP] E′1...E′ k E [SEP] , where E [CLS] is the embedding vector of [CLS], E1 is the embedding vector of t1, E j t j The embedding vector, E [SEP] is the embedding vector of [SEP], E′1 is the embedding vector of p1, and E′ k For p k The embedding vector, E [SEP] is the embedding vector of [SEP]; for the context and punchline in the embedding vector matrix, different segment embeddings are added to obtain the first embedding representation. Position embedding is added to each word in the first embedding representation to obtain the final text vector embedding representation. The position embedding represents the position of the word in the first vector embedding representation. Step 2.3: Input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t .

3. A multimodal humor recognition method according to claim 2, characterized in that: Step 5 specifically includes: Step 5.1: For each input video data in step 1, extract the contextual humor features and the humor features of the punchline from the input text, and convert the contextual humor features and the humor features of the punchline into a form to obtain the converted humor features X f =[PAD]C f [SEP]P f , where C f is the humor feature corresponding to the context, P f is the humor feature corresponding to the punchline, [PAD] is a placeholder, and [SEP] is used to convert C f With P f separate; Step 5.2: Use the P2FA forced alignment model to extract the time corresponding to each word in the text segmented in step 2.1, and then perform face image feature f according to the time corresponding to each word. v and audio features f a Segmentation is performed and the segmented face image features are represented as X v =[PAD]C v [SEP]P v , C v is the facial image feature corresponding to the context, P v is the facial image feature corresponding to the laugh point, and the segmented audio feature is represented as X a =[PAD]C a [SEP]P a , C a is the audio feature corresponding to the context, P a The audio features corresponding to the funny points; Step 5.3: The segmented face image features X v Input Transformer encoder to obtain visual modality feature U v , specifically expressed by the following formula: U v =TransformerEncoder(X v ); The segmented audio feature X a Input Transformer encoder to obtain audio modality feature U a , specifically expressed by the following formula: U a =TransformerEncoder(X a ); Set humor feature X f Input Transformer encoder to get humor modality feature U f , specifically expressed by the following formula: U f =TransformerEncoder(X f )。 4. The multimodal humor recognition method according to claim 1, characterized in that: Step A specifically includes: Step A1: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f , respectively input into the linear layer to obtain the final text modality feature U of the same dimension t ′, the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U f ′, the formula of the linear layer is expressed as: Y=XW1+b1; Among them, W1 is the learning weight matrix and b1 is the bias vector; Specifically, the text modality feature U t , visual modality features U v , audio modal features U a and humor modal features into X to obtain the final text modal features U t ′, the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U f '; Step A2: Final text modality feature U t ′, the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U f ' to splice and get the matrix M=[U t ′,U′ v ,U′ a ,U f ′], through the multi-head self-attention mechanism, the matrix M is processed to obtain M′=[U t ″,U′ v ′,U′ a ′,U f ″], which is specifically achieved through the following formula: M′M=MultiHead(M,θ)=Concat(head1,...,head h )W o ; θ={W q ,W k ,W v ,W o ); Among them, MultiHead represents multi-head splicing, head1 represents the first attention head, head h represents the hth attention head, W q ,W k ,W v ,W o is the attention mechanism parameter, where each attention head is expressed as: M=K=V=Q; Among them, head c represents the cth attention head, Attention represents the self-attention mechanism, is the attention mechanism parameter; Replace U in M′ t ″,U′ v ′,U′ a ′,U f ″ to combine and obtain the joint vector U final , specifically expressed by the following formula: The joint vector U final The input is fed into the fully connected layer and the predicted probability distribution P representing the humor polarity is obtained through the Softmax function. It is specifically expressed by the following formula: P=SoftMax(U final W2+b2); Among them, W2 and b2 are the parameters of the fully connected layer.

5. The multimodal humor recognition method according to claim 1, characterized in that: Step B specifically includes: Step B1: Use graph attention network to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Update to get the updated text modal feature f t , updated humor modal features f f , updated visual modality features f v and the updated audio modality features f a ; Step B2: Through the cross attention mechanism, the updated text modality feature f t and the updated humor modal feature f f Perform modal fusion to obtain the first fusion feature f t,f , specifically expressed by the following formula: Through the cross attention mechanism, the updated visual modality feature f v and the updated audio modality features f a Perform modal fusion to obtain the second fusion feature f a,v , specifically expressed by the following formula: Step B3: respectively perform the first fusion feature f t,f and the second fusion feature f a,v Perform linear transformation to obtain the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is implemented by the following formula: in, and is the weight matrix, K1 is the first fusion feature f t,f The key feature matrix, V1 is the first fusion feature f t,f The feature matrix of the value of K2 is the second fusion feature f a,v The key feature matrix, V2 is the second fusion feature f a,v The characteristic matrix of the values ​​of ; Step B4: Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, and the outputs of all attention heads are aggregated to obtain the first output O1 and the second output O2; Step B5: Combine O1 with the first fusion feature f t,f Connect and get the updated modal feature f′ t,f , O2 is combined with the second fusion feature f a,v Connect and get the updated modal feature f′ a,v , specifically expressed by the following formula: f′ t,f =O1+f t,f ; f′ a,v =O2+f a,v ; Step B6: f′ t,f and f′ a,v Input into the multi-head self-attention layer in sequence to obtain the processed information feature f a,v,f,t , input the processed information features into the maximum pooling layer to obtain the final feature f final , the final feature f final Input the softmax layer to obtain the predicted probability distribution P that represents the humor polarity of the input video data, which is specifically expressed by the following formula: P=softmax(f final W5+b5); Among them, W5 and b5 are the parameters of the softmax layer.

6. The multimodal humor recognition method according to claim 5, characterized in that: Step B1 specifically includes: The text modality features U t , visual modality features U v , audio modal features U a and humor modality characteristics U f Perform feature extraction and dependency analysis to obtain a node feature matrix for text mode, a node feature matrix for visual mode, a node feature moment for audio mode, a node feature matrix for humor mode, and an adjacency matrix. The adjacency matrix represents the grammatical dependency relationship between words. The node feature matrix of the text modality includes a text feature vector corresponding to each word, and each text feature vector is used as a node of a text graph to obtain an initial text graph; the node feature matrix of the visual modality includes a visual feature vector corresponding to each word, and each visual feature vector is used as a node of a visual graph to obtain an initial visual graph; the node feature matrix of the audio modality includes an audio feature vector corresponding to each word, and each audio feature vector is used as a node of an audio graph to obtain an initial audio graph; the node feature matrix of the humor modality includes a humor feature vector corresponding to each word, and each humor feature vector is used as a node of a humor graph to obtain an initial humor graph; The weight W3 of the linear transformation is initialized through the uniform distribution formula. For each node in the initial text graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix. The initialized linear transformation weight W3 is multiplied with the text feature vector of the node and the text feature vector of an adjacent node respectively. The result of the multiplication with the text feature vector of the node and the result of the multiplication with the text feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes. This is specifically achieved through the following formula: <h2 style=";text-align:left;direction:ltr">e<h2 style=";text-align:left;direction:ltr"> mn <h2 style=";text-align:left;direction:ltr"> =a([W3A<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> ||W3A<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ]),n∈N<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> ; Among them, e mn is the correlation coefficient between node m and node n, || represents the connection operation, a(·) represents the mapping operation, A m is the text feature vector corresponding to node m, A n is the text feature vector corresponding to node n, N m Represents the subscript set of nodes connected to node m; Normalize the correlation coefficient between the node and its adjacent nodes to obtain the normalized attention coefficient, which is specifically achieved through the following formula: Among them, α mn is the normalized attention coefficient of node m and node n, e mr is the correlation coefficient between node m and node r; The normalized attention coefficients are weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated text modality feature f t , which is specifically achieved through the following formula: Among them, A′ m (l) represents the output vector of node m in the lth attention head, that is, the updated text modality feature f t , L represents the total number of attention heads, σ(·) represents the activation function, represents the normalized attention coefficient of the lth attention head, represents the weight matrix of the lth attention head; Similarly, for each node in the initial visual graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the visual feature vector of the node and the visual feature vector of an adjacent node are multiplied by the initialized linear transformation weight W3 respectively, and the result of multiplying the visual feature vector of the node and the result of multiplying the visual feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated visual modal feature f v ; For each node in the initial audio graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the audio feature vector of the node and the audio feature vector of an adjacent node, and the result of multiplication with the audio feature vector of the node and the result of multiplication with the audio feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node, and the correlation coefficient between the node and the adjacent node is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated audio modal feature f a ; For each node in the initial humor graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the humor feature vector of the node and the humor feature vector of an adjacent node, and the result of multiplying with the humor feature vector of the node and the result of multiplying with the humor feature vector of the adjacent node is spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes. The correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed. The result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated humor modal feature f f .

7. The multimodal humor recognition method according to claim 5, characterized in that: Step B4 specifically includes: Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, which is implemented by the following formula: Among them, d k Represents the dimension of the key, Attention h (Q1, K2, V2) is the output of the hth attention head after multi-head attention calculation of Q1, K2 and V2. h (Q2, K1, V1) is the output of the hth attention head after multi-head attention calculation of Q2, K1 and V1; H represents the total number of attention heads, is the index of the h-th attention head in Q1; is the index of the h-th attention head in Q2; is the index of the h-th attention head in K2; for The index of the h-th attention head in ; is the index of the h-th attention head in V1; is the index of the h-th attention head in V2; for The transpose of for The transpose of Aggregate the outputs of all attention heads to get, which is achieved by the following formula: O1=Concat([Attention 1 (Q1,K2,V2),...,Attention H (Q1,K2,V2)]); O2=Concat([Attention 1 (Q2,K1,V1),...,Attention H (Q2,K1,V1)]); Among them, H represents the total number of attention heads, O1 is the output of all attention heads obtained by multi-head attention calculation of Q1 with K2 and V2, and O2 is the output of all attention heads obtained by multi-head attention calculation with K1 and V1.

Images