Humor identification method oriented to multiple modes

Abstract

The invention provides a humor recognition method oriented to multiple modes, and relates to the technical field of multi-mode humor recognition, in one implementation mode, introduced external knowledge is used as an independent mode, and the independent mode is fused with text, audio and video modes. According to the method, dimension alignment is carried out through a full connection layer, then the dimension alignment is spliced and input into Transform to learn an interaction relationship among four modals, and at the same time, modal fusion is optimized by means of a cross attention mechanism, so that the problems of external knowledge introduction and modal fusion schemes are solved. In another implementation mode, data enhancement is carried out through a graph attention network to make up for the problem of scarcity of the number of current data sets, related modes are fused and spliced, then are sent into a cross attention network, are further fused through a multi-head self-attention layer and are finally input into a maximum pool layer to obtain single feature representation of a fusion vector, and the single feature representation of the fusion vector is obtained. And prediction probability distribution of humor polarity is obtained, so that more effective modal fusion is obtained.

Description

Technical Field

[0001] The invention relates to the technical field of multimodal humor recognition, in particular to a multimodal humor recognition method. Background Art

[0002] Early research on humor was mainly limited to text data, using machine learning methods to identify humor, and identifying humor in text by constructing feature vectors and training classifiers. With the popularity of neural networks, many traditional machine learning methods have gradually been abandoned, replaced by some deep learning structures, such as the convolutional neural network used by Chen and Lee, and the Transformer architecture used by Weller and Seppi. In the humorous and offensive text recognition and evaluation task held by the International Association for Computational Linguistics in 2021, Xie et al. used the decoupled attention decoding enhanced BERT model DeBERTa to predict humor tags, Samuel Akrah used the pre-trained RoBERTa model to complete the task, and BoHuang et al. used a model that combined ALBERT, CNN, and Tf-Idf algorithms for humor recognition.

[0003] With the enrichment of video and audio content, researchers have gradually focused on how to effectively fuse information from different modalities (such as text, speech, images, etc.) to improve the accuracy and robustness of humor recognition. The bimodal modal fusion network proposed by Wei Han et al. is an end-to-end model that can effectively fuse and separate data from different modalities. This method helps to process the associations and differences between modalities in multimodal data, thereby achieving more accurate humor recognition.

[0004] In addition, the Transformer model based on rich humor knowledge published in AAAI is one of the important advances in this field. This study introduced external knowledge (such as ambiguity theory and superiority theory) into multimodal humor recognition for the first time, providing deeper background information for humor understanding, greatly expanding the expressive power and recognition accuracy of existing models. The optimal transmission method has been applied in the research of Shraman Pramanick et al. to solve the problem of limited humor data. Through this method, the model can better handle humor data from different sources and types, and improve recognition performance in data-scarce conditions. In 2023, Wu et al. proposed a modal fusion model based on the attention mechanism. First, the information of each modality is independently encoded, and then the interaction information between different modalities is combined through the attention mechanism. This method can dynamically focus on the most important parts of humor recognition in different modalities, thereby enhancing the performance of the model. In addition, the emotion knowledge enhanced self-supervised learning method proposed by Fan et al. introduced emotion knowledge and non-verbal behavior information into multimodal humor recognition. Through the training of the self-supervised learning model, it can effectively learn the emotional characteristics and non-verbal behavior cues of humor without relying on a large amount of labeled data, making the model more natural and close to real language usage scenarios when processing humor.

[0005] Although the exploration of humor recognition has been extensive, the field of humor recognition on multimodal data still faces challenges.

[0006] First, external knowledge can help the model understand the context and identify humorous elements such as puns and metaphors, thereby improving the accuracy of recognition. However, in current work, the introduction of external knowledge is lacking. Secondly, the commonly used methods of multimodal fusion methods currently use weighted addition, splicing, or averaging, which can easily cause information redundancy or loss, thereby affecting the experimental results. Previous fusion schemes did not consider the relationship between different modalities. Finally, the two currently available benchmark datasets MUStARD and UR-FUNNY have relatively small data volumes, which will result in the experiment being unable to accurately capture important features or information in the data during training, and may even cause overfitting and poor generalization. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention aims to propose a multimodal humor recognition method, comprising:

[0008] Step 1: Acquire a plurality of input video data, wherein the input video data includes input text, face image and audio information;

[0009] Step 2: For each input video data in step 1, perform word segmentation and vector embedding on the input text to obtain the final text vector, and input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t ;

[0010] Step 3: For each input video data in step 1, align and calibrate the face image by affine transformation technology to obtain an aligned face image; crop the face part in the aligned face image to obtain a first image, input the first image into a deep convolutional neural network CNN to obtain a face feature vector, perform L2 normalization on the face feature vector to obtain a normalized face feature vector, and select action units and face shape parameter features from the normalized face feature vector as the final face image feature f v ;

[0011] Step 4: For the audio information in step 1, extract the pitch information from the audio information, extract the pitch information through COVAREP to obtain multiple pitch features, and process the multiple pitch features through SPS software to obtain the audio feature f after dimensionality reduction a ;

[0012] Step 5: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and then extract the humorous features U according to the humorous features and the text modality features U t , the final face image feature f v and audio feature f a , get the visual modality feature U v , audio modal features U a and humor modality feature U f ;

[0013] Step 6: Text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Processing is performed to obtain a predicted probability distribution P that represents humor polarity, which is specifically achieved through step A or step B:

[0014] Step A: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Input the linear layer to obtain the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ aand the final humor modal feature U′ f , for the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f The spliced ​​results are processed through a multi-head self-attention mechanism and a fully connected layer to obtain a predicted probability distribution P that represents the humor polarity of the input video data;

[0015] Step B: Attention network is used to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Update, and pass the updated features through the cross-attention mechanism, linear transformation, multi-head cross-attention mechanism and multi-head self-attention layer to obtain the predicted probability distribution P that characterizes the humor polarity of the input video data;

[0016] Step 7: Use the predicted probability distribution P as the predicted value, and obtain the true value representing the humor polarity corresponding to the input video data. Calculate the loss function value based on the predicted value and the true value, which is specifically implemented by the following formula:

[0017]

[0018] Among them, BinaryCrossEntropy is the loss function value, N is the number of input video data, y i is the true value of the i-th input video data, y′ i is the predicted value of the i-th input video data;

[0019] Through the loss function value, the embedding vector matrix, deep convolutional neural network CNN, Transformer encoder, linear layer parameters W1 and b1, and fully connected layer parameters W2 and b2 are updated until the loss function value is less than the preset threshold. At this time, the output predicted probability distribution P is the humor polarity prediction result of the input video data.

[0020] Optionally, step 2 specifically includes:

[0021] Step 2.1: For each input video data in step 1, the input text is segmented by the WordPiece word segmenter of the ALBERT model to obtain the segmented text S′={(t1, t2, ..., t j ),(p1,p2,...,p k )}, where (t1, t2, ..., t j) is the context of the input text, t1 represents the first word in the context, t2 represents the second word in the context, and t j represents the jth word in the context, j represents the total number of subwords in the context, (p1, p2, ..., p k ) is the punchline of the input text, p1 represents the first word of the punchline, p2 represents the second word of the punchline, and p k The kth word representing the punchline, k is the total number of subwords of the punchline, and tags are added to the segmented text. Specifically, a [CLS] tag is added to the beginning of the segmented text, a [SEP] tag is added between the context and the punchline, and a [SEP] tag is added to the end of the segmented text to obtain a text sequence X = [CLS]t1...t j [SEP]p1...p k [SEP];

[0022] Step 2.2: Convert the text sequence into an embedded vector and obtain the embedded vector matrix X1 = E [CLS] E1...E j E [SEP] E′1...E′ k E [SEP] , where E [CLS] is the embedding vector of [CLS], E1 is the embedding vector of t1, E j t j The embedding vector, E [SEP] is the embedding vector of [SEP], E′1 is the embedding vector of p1, and E′ k For p k The embedding vector, E [SEP] is the embedding vector of [SEP]; for the context and jokes in the embedding vector matrix, different segment embeddings are added to obtain the first embedding representation, and the position embedding is added to each word in the first embedding representation to obtain the final text vector embedding representation, where the position embedding represents the position of the word in the first vector embedding representation;

[0023] Step 2.3: Input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t .

[0024] Optionally, step 5 specifically includes:

[0025] Step 5.1: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and convert the humorous features of the context and the humorous features of the jokes to obtain the converted humorous features X f =[PAD]C f [SEP]Pf , where C f is the humorous feature corresponding to the context, P f is the humor feature corresponding to the punchline, [PAD] is a placeholder, and [SEP] is used to convert C f With P f separate;

[0026] Step 5.2: Use the P2FA forced alignment model to extract the time corresponding to each word in the segmented text described in step 2.1, and align the facial image feature f according to the time corresponding to each word. v and audio feature f a Segmentation is performed, and the segmented face image features are represented as X v =[PAD]C v [SEP]P v , C v is the facial image feature corresponding to the context, P v is the facial image feature corresponding to the laugh point, and the segmented audio feature is represented as X a =[PAD]C a [SEP]P a , C a is the audio feature corresponding to the context, P a The audio features corresponding to the laugh points;

[0027] Step 5.3: The segmented face image feature X v Input Transformer encoder to get visual modality feature U v , which is specifically expressed by the following formula:

[0028] U v = TransformerEncoder(X v );

[0029] The segmented audio feature X a Input Transformer encoder to get audio modality feature U a , which is specifically expressed by the following formula:

[0030] U a = TransformerEncoder(X a );

[0031] Add humor feature X f Input Transformer encoder to get humor modality feature U f , which is specifically expressed by the following formula:

[0032] U f= TransformerEncoder(X f ).

[0033] Optionally, step A specifically includes:

[0034] Step A1: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f , respectively input into the linear layer to obtain the final text modality feature U′ of the same dimension t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f , the formula of the linear layer is expressed as:

[0035] Y=XW1+b1;

[0036] Among them, W1 is the learning weight matrix, b1 is the bias vector;

[0037] Specifically, the text modality feature U t , visual modality features U v , audio modal features U a and humor modality features into X to obtain the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f ;

[0038] Step A2: Final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f Splice and get the matrix M = [U′ t ,U′ v ,U′ a ,U′ f ], through the multi-head self-attention mechanism, the matrix M is processed to obtain M′=[U″ t ,U″ v ,U″ a ,U″ f ], which is implemented by the following formula:

[0039] M=MultiHead(M,θ)=Concat(head1,...,head h )Wo ;

[0040] θ={W q ,W k ,W v ,W o );

[0041] Among them, MultiHead represents multi-head splicing, head1 represents the first attention head, and head h represents the hth attention head, W q ,W k ,W v ,W o is the attention mechanism parameter, where each attention head is expressed as:

[0042]

[0043] M = K = V = Q;

[0044] Among them, head c represents the cth attention head, Attention represents the self-attention mechanism, is the attention mechanism parameter;

[0045] Replace U″ in M′ t ,U″ v ,U″ a ,U″ f Combine and get the joint vector U final , which is specifically expressed by the following formula:

[0046]

[0047] The joint vector U final Input the fully connected layer and use the Softmax function to obtain the predicted probability distribution P that represents the humor polarity, which is specifically expressed by the following formula:

[0048] P = SoftMax(U final W2+b2);

[0049] Among them, W2 and b2 are the parameters of the fully connected layer.

[0050] Optionally, step B specifically includes:

[0051] Step B1: Use graph attention network to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Update to get the updated text modal feature f t, the updated humor modal feature f f , updated visual modality features f v and the updated audio modal features f a ;

[0052] Step B2: Use the cross-attention mechanism to update the text modality feature f t and the updated humor modal feature f f Perform modal fusion to obtain the first fusion feature f t,f , which is specifically expressed by the following formula:

[0053]

[0054] Through the cross attention mechanism, the updated visual modality feature f v and the updated audio modal features f a Perform modal fusion to obtain the second fusion feature f a,v , which is specifically expressed by the following formula:

[0055]

[0056] Step B3: For the first fusion feature f t,f and the second fusion feature f a,v Perform linear transformation to obtain the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is implemented by the following formula:

[0057]

[0058]

[0059] in, and is the weight matrix, K1 is the first fusion feature f t,f The key feature matrix, V1 is the first fusion feature f t,f The feature matrix of the value of, K2 is the second fusion feature f a,v The key feature matrix, V2 is the second fusion feature f a,v The feature matrix of the values ​​of ;

[0060] Step B4: Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, and the outputs of all attention heads are aggregated to obtain the first output O1 and the second output O2;

[0061] Step B5: Combine O1 with the first fusion feature f t,f Connect and get the updated modal feature f′ t,f , O2 is combined with the second fusion feature f a,v Connect and get the updated modal feature f′ a,v , which is specifically expressed by the following formula:

[0062] f′ t,f =O1+f t,f ;

[0063] f′ a,v =O2+f a,v ;

[0064] Step B6: f′ t,f and f′ a,v Input into the multi-head self-attention layer in sequence to obtain the processed information feature f a,v,f,t , input the processed information features into the maximum pooling layer to obtain the final feature f final , the final feature f final Input the softmax layer to obtain the predicted probability distribution P that represents the humor polarity of the input video data, which is specifically expressed by the following formula:

[0065] P = softmax(f final W5+b5);

[0066] Among them, W5 and b5 are the parameters of the softmax layer.

[0067] Optionally, step B1 specifically includes:

[0068] The text modality features U t , visual modality features U v , audio modal features U a and humor modality feature U f Perform feature extraction and dependency analysis to obtain the node feature matrix of the text mode, the node feature matrix of the visual mode, the node feature matrix of the audio mode, the node feature matrix of the humor mode, and the adjacency matrix, where the adjacency matrix represents the grammatical dependency relationship between words;

[0069] Among them, the node feature matrix of the text mode includes the text feature vector corresponding to each word, and each text feature vector is used as a node of the text graph to obtain an initial text graph; the node feature matrix of the visual mode includes the visual feature vector corresponding to each word, and each visual feature vector is used as a node of the visual graph to obtain an initial visual graph; the node feature matrix of the audio mode includes the audio feature vector corresponding to each word, and each audio feature vector is used as a node of the audio graph to obtain an initial audio graph; the node feature matrix of the humor mode includes the humor feature vector corresponding to each word, and each humor feature vector is used as a node of the humor graph to obtain an initial humor graph;

[0070] The weight W3 of the linear transformation is initialized through the uniform distribution formula. For each node in the initial text graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix. The weight W3 of the linear transformation is initialized and multiplied with the text feature vector of the node and the text feature vector of an adjacent node respectively. The result of multiplying the text feature vector of the node and the result of multiplying the text feature vector of the adjacent node are concatenated, and the concatenated result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node. This is specifically implemented by the following formula:

[0071] e mn =a([W3A m ||W3A n ]),n∈N m ;

[0072] Among them, e mn is the correlation coefficient between node m and node n, || represents the connection operation, a(·) represents the mapping operation, A m is the text feature vector corresponding to node m, A n is the text feature vector corresponding to node n, N m Represents the subscript set of nodes connected to node m;

[0073] The correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, which is specifically implemented by the following formula:

[0074]

[0075] Among them, α mn is the normalized attention coefficient of node m and node n, e mr is the correlation coefficient between node m and node r;

[0076] The normalized attention coefficients are weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated text modality feature f t , which is specifically achieved through the following formula:

[0077]

[0078] Among them, A′ m (l) represents the output vector of node m at the lth attention head, i.e., the updated text modality feature f t , L represents the total number of attention heads, σ(·) represents the activation function, represents the normalized attention coefficient of the lth attention head, represents the weight matrix of the lth attention head;

[0079] Similarly, for each node in the initial visual graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the visual feature vector of the node and the visual feature vector of an adjacent node are multiplied by the weight W3 of the initialized linear transformation respectively, and the result of multiplying the visual feature vector of the node and the result of multiplying the visual feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated visual modal feature f v ;

[0080] For each node in the initial audio graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the audio feature vector of the node and the audio feature vector of an adjacent node, and the result of multiplying the audio feature vector of the node and the result of multiplying the audio feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node, and the correlation coefficient between the node and the adjacent node is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated audio modal feature f a ;

[0081] For each node in the initial humor graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the weight W3 of the initialized linear transformation is respectively multiplied with the humor feature vector of the node and the humor feature vector of an adjacent node, and the result of multiplying the humor feature vector of the node and the result of multiplying the humor feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated humor modal feature f f .

[0082] Optionally, step B4 specifically includes:

[0083] Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, which is implemented by the following formula:

[0084]

[0085] Among them, d k Indicates the dimension of the key, Attention h (Q1, K2, V2) is the output of the hth attention head after multi-head attention calculation of Q1, K2 and V2. h (Q2, K1, V1) is the output of the hth attention head after multi-head attention calculation of Q2, K1 and V1; H represents the total number of attention heads, is the index of the h-th attention head in Q1; is the index of the h-th attention head in Q2; is the index of the h-th attention head in K2; for The index of the h-th attention head in ; is the index of the h-th attention head in V1; is the index of the h-th attention head in V2; for The transpose of for The transpose of

[0086] Aggregate the outputs of all attention heads to get, which is implemented by the following formula:

[0087] O1=Concat([Attention 1 (Q1,K2,V2),...,Attention H (Q1,K2,V2)]);

[0088] O2=Concat([Attention 1 (Q2,K1,V1),...,Attention H (Q2,K1,V1)]);

[0089] Among them, H represents the total number of attention heads, O1 is the concatenated output of all attention heads after Q1 performs multi-head attention calculation with K2 and V2, and O2 is the concatenated output of all attention heads after multi-head attention calculation with K1 and V1.

[0090] The beneficial effects of adopting the above technical solution are:

[0091] In one implementation, the present invention treats the introduced external knowledge as a separate modality and fuses it with text, audio and video modalities. First, the dimensions of different modal features are aligned through the fully connected layer, and then they are spliced ​​and input into the Transformer to learn the interactive relationship between the four modalities and output a single feature representation. The present invention optimizes modal fusion with the help of the cross-attention mechanism, solving the problem of external knowledge introduction and modal fusion scheme. In another implementation, the present invention uses a graph attention network to perform data enhancement to make up for the scarcity of the current data set, and then considers the relationship between the four modalities. It is believed that the humor-related modality is closely related to the text modality, and the audio and visual modalities are closely related, so the related modalities are fused and spliced. Then it is sent to the cross-attention network to learn the complementary representation of the features and output, and then further fused through the multi-head self-attention layer, and finally input into the maximum pooling layer to obtain a single feature representation of the fusion vector, thereby obtaining a more effective modal fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 is a flow chart of a multi-modal humor recognition method in an embodiment of the present invention;

[0093] Figure 2 A schematic diagram of a process of obtaining a predicted probability distribution P in an embodiment of the present invention;

[0094] Figure 3 A schematic diagram of another process of obtaining a predicted probability distribution P in an embodiment of the present invention. DETAILED DESCRIPTION

[0095] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0096] In view of the problems existing in the prior art, the present invention provides a humor recognition method for multimodality. In view of the two problems of how to introduce external knowledge and how to integrate external knowledge, the present invention proposes a humor recognition method based on HTT. In order to introduce external knowledge into the humor recognition model, the currently recognized mainstream humor theory is summarized and summarized as the theoretical basis for the acquisition of humor external knowledge. The present invention selects appropriate features for encoding representation as the injection of external knowledge. In order to create an effective modality fusion solution, the present invention treats the introduced external knowledge as a separate modality and integrates it with text, audio and video modalities.

[0097] Specifically, the present invention solves the problem of external knowledge introduction and modal fusion scheme by proposing a humor recognition method based on HTT. In order to introduce external knowledge into the humor recognition model, the present invention summarizes the currently recognized mainstream humor theories as the theoretical basis for external knowledge acquisition. There are three theories finally adopted: superiority theory from the perspective of social behavior, release theory from the perspective of psychological analysis, and incongruity theory from the perspective of psychological cognition. Taking the release theory as the starting point, combined with the data set provided by SemEval 2021Task 7 and the Word Art Creator word cloud tool, an offensive dictionary is constructed to calculate the offensive level of each word as a humor feature; taking the superiority theory as the starting point, the SentiWordNet sentiment dictionary is used to calculate the sentiment score of each word as a humor feature; finally, two humor-related features are constructed based on the incongruity theory. The first is to calculate ambiguity. Specifically, the number of semantic sets of each word is calculated through WordNet, and the top T words are selected as candidate ambiguous words, and then the cosine similarity function is selected to measure the semantic distance, and then the standard deviation of these similarity values ​​is calculated and the standard deviation is used as a measure of ambiguity. The second is to use the NRC VAD sentiment dictionary to extract the sentiment polarity score of each word. The present invention encodes the four features mentioned above as the injection of external knowledge. In order to create an effective modal fusion solution, the present invention treats the introduced external knowledge as a separate modality and fuses it with text, audio and video modalities. First, the dimensions of different modal features are aligned through the fully connected layer, and then they are spliced ​​and input into the Transformer to learn the interaction between the four modalities and output a single feature representation.

[0098] In order to solve the problems of data shortage and modal fusion, a humor recognition method based on GACA is proposed. This paper proposes a scheme to enhance data by using graph attention network, in which data is updated to obtain enhanced feature data. This paper optimizes modal fusion with the help of cross-attention mechanism, fully considers the relationship between the four modalities, and effectively fuses and splices related modalities.

[0099] Specifically, the present invention solves the problem of data shortage and modal fusion scheme by proposing a humor recognition method based on GACA. In order to make up for the problem of the scarcity of the current data set, the present invention proposes a scheme for data enhancement using a graph attention network. Specifically, for the text modality, each word is regarded as a data point, and then an adjacency matrix is ​​constructed according to the dependency relationship between the words, and then the vector representation of the word is obtained by Glove embedding to obtain a point feature matrix. For the other three modalities, because they have been aligned with the text words in the data processing stage, the adjacency matrix and the text are consistent, and the point feature matrix is ​​obtained by the corresponding feature extraction tool. Finally, the two matrices are sent together to the graph attention network to complete the update of the data and obtain the enhanced feature data. In order to create a more effective modal fusion scheme, the present invention optimizes modal fusion with the help of a cross-attention mechanism. First, the relationship between the four modalities is considered. It is believed that the humor-related modality is closely related to the text modality, and the audio and visual modalities are closely related, so the related modalities are fused and spliced. It is then sent to the cross-attention network to learn the complementary representation of the features and output them, and then further fused through the multi-head self-attention layer, and finally input into the maximum pooling layer to obtain a single feature representation of the fused vector.

[0100] Based on the above scheme, the present invention provides a multi-modal humor recognition method, combined with Figure 1 , which may specifically include the following steps:

[0101] Step 1: Acquire a plurality of input video data, wherein the input video data includes input text, face image and audio information;

[0102] Step 2: For each input video data in step 1, perform word segmentation and vector embedding on the input text to obtain the final text vector, and input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t ;

[0103] Step 2.1: For each input video data in step 1, the input text is segmented by the WordPiece word segmenter of the ALBERT model to obtain the segmented text S′={(t1, t2, ..., t j ),(p1,p2,...,p k)}, where (t1, t2, ..., t j ) is the context of the input text, t1 represents the first word in the context, t2 represents the second word in the context, and t j represents the jth word in the context, j represents the total number of subwords in the context, (p1, p2, ..., p k ) is the punchline of the input text, p1 represents the first word of the punchline, p2 represents the second word of the punchline, and p k The kth word representing the punchline, k is the total number of subwords of the punchline, and tags are added to the segmented text. Specifically, a [CLS] tag is added at the beginning of the segmented text, and a [SEP] tag is added between the context and the punchline to separate the two information sources, the context and the punchline, so as to model the context and the punchline. A [SEP] tag is added at the end of the segmented text to obtain a text sequence X'=[CLS]t1...t j [SEP]p1...p k [SEP];

[0104] Step 2.2: Convert the text sequence into an embedded vector and obtain the embedded vector matrix X1 = E [CLS] E1...E j E [SEP] E′1...E′ k E [SEP] , where E [CLS] is the embedding vector of [CLS], E1 is the embedding vector of t1, E j t j The embedding vector, E [SEP] is the embedding vector of [SEP], E′1 is the embedding vector of p1, and E′ k For p k The embedding vector, E [SEP] is the embedding vector of [SEP]; different segment embeddings are added to the context and punchline in the embedding vector matrix to obtain the first embedding representation. Adding different segment embeddings helps the model recognize and correctly process two different text inputs; position embedding is added to each word in the first embedding representation to obtain the final text vector embedding representation. Position embedding represents the position of a word in the first vector embedding representation. Adding position embedding can help the model learn contextual relationships and thus better understand the semantics of sentences;

[0105] Step 2.3: Input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t .

[0106] Specifically, the final text vector embedding representation is input into the ALBERT model, and the input is processed by the Encoder module in the multi-layer stacked Transformer structure. The Encoder module of each layer uses a multi-head self-attention mechanism to capture the global dependencies between different positions in the input sequence and perform context modeling on the text semantic information, thereby obtaining high-quality text modality features U containing rich semantic information. t .

[0107] Step 3: For each input video data in step 1, since the face in the face image may be tilted, the face needs to be aligned and corrected before the image is input into the convolutional neural network. Therefore, the present invention aligns and calibrates the face image through affine transformation technology to obtain an aligned face image;

[0108] Specifically, facial key points are detected on the face image. These key points are usually located at the corners of the eyes, nose, and facial contours. Then, the key points and affine transformation technology are used to align and calibrate the face to adjust the face to the correct position, thereby reducing errors caused by different postures.

[0109] The face part of the aligned face image is cut out to obtain a first image, and the first image is input into a deep convolutional neural network (CNN) to obtain a face feature vector, wherein the deep convolutional neural network (CNN) is composed of a batch input layer and a deep convolutional neural network (CNN); the face feature vector is L2 normalized to obtain a normalized face feature vector, and the L2 normalization is used to map the feature to a hyperspherical Euclidean space to ensure that the feature vector has a unit length; in the normalized face feature vector, the action unit and the face shape parameter feature are selected as the final face image feature f v ;

[0110] Step 4: For the audio information in step 1, extract the pitch information from the audio information, extract the pitch information through COVAREP, and obtain a plurality of pitch features. Specifically, the pitch information can be extracted through the methods of spectrum envelope estimation, beam tracking, sinusoidal modeling, phase processing and glottal analysis in COVAREP. The obtained plurality of pitch features include a series of features such as Melcepstral coefficient, fundamental frequency, voiced / unvoiced segment, normalized amplitude quotient, quasi-open quotient, glottal source parameter, harmonic model and phase distortion, formant, etc.; since some features such as cepstrum features and glottal features show a high correlation in some analysis aspects, the present invention processes the plurality of pitch features through SPS software to obtain the audio feature f after dimensionality reduction. a ;

[0111] Step 5: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and then extract the humorous features U according to the humorous features and the text modality features U t , the final face image feature f v and the audio feature f a , get the visual modality feature U v , audio modal features U a and humor modality feature U f ;

[0112] Step 5.1: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and convert the humorous features of the context and the humorous features of the jokes to obtain the converted humorous features X f =[PAD]C f [SEP]P f , where C f is the humorous feature corresponding to the context, P f is the humor feature corresponding to the punchline, [PAD] is a placeholder, and [SEP] is used to convert C f With P f separate;

[0113] Step 5.2: Use the P2FA forced alignment model to extract the time corresponding to each word in the segmented text described in step 2.1, and align the facial image feature f according to the time corresponding to each word. v and audio feature f a Segmentation is performed, and the segmented face image features are represented as X v =[PAD]C v [SEP]P v , C v is the facial image feature corresponding to the context, P v is the facial image feature corresponding to the laugh point, and the segmented audio feature is represented as X a =[PAD]C a [SEP]P a , C a is the audio feature corresponding to the context, P a The audio features corresponding to the laugh points;

[0114] Among them, humor feature X f , segmented face image features X v And the segmented audio features X a It can be uniformly expressed as X E , where E = {a, v, f}, X E ∈R T×dE , d Eis the dimension of the corresponding feature, T represents the number of words;

[0115] Step 5.3: The segmented face image feature X v Input Transformer encoder to get visual modality feature U v , which is specifically expressed by the following formula:

[0116] U v = TransformerEncoder(X v );

[0117] The segmented audio feature X a Input Transformer encoder to get audio modality feature U a , which is specifically expressed by the following formula:

[0118] U a = TransformerEncoder(X a );

[0119] Add humor feature X f Input Transformer encoder to get humor modality feature U f , which is specifically expressed by the following formula:

[0120] U f = TransformerEncoder(X f ).

[0121] Among them, the visual modality feature U v , audio modal features U a and humor modality feature U f It can be uniformly expressed as U E , d UE The output dimension of the Transformer encoder corresponding to the modality feature E is the dimension of the feature in the corresponding modality.

[0122] Step 6: Text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Processing is performed to obtain a predicted probability distribution P that represents humor polarity, which is specifically achieved through step A or step B:

[0123] The present invention provides a humor recognition method based on HTT, which is specifically implemented through step A.

[0124] Step A: Transform the text modality feature U t , visual modality features Uv , audio modal features U a and humor modality feature U f Input the linear layer to obtain the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f , for the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f The spliced ​​results are processed through a multi-head self-attention mechanism and a fully connected layer to obtain a predicted probability distribution P that represents the humor polarity of the input video data;

[0125] Combination Figure 2 , which is achieved through the following steps:

[0126] Step A1: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f , respectively input into the linear layer to obtain the final text modality feature U′ of the same dimension t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f , the formula of the linear layer is expressed as:

[0127] Y=XW1+b1;

[0128] Among them, W1 is the learning weight matrix, b1 is the bias vector;

[0129] Specifically, the text modality feature U t , visual modality features U v , audio modal features U a and humor modality features into X to obtain the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f ;

[0130] Step A2: Final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′a and the final humor modal feature U′ f Splice and get the matrix M = [U′ t ,U′ v ,U′ a ,U′ f ], through the multi-head self-attention mechanism, the matrix M is processed to obtain M′=[U″ t ,U″ v ,U″ a ,U″ f ], which is implemented by the following formula:

[0131] M′=MultiHead(M,θ)=Concat(head1,...,head h )W o ;

[0132] θ={W q ,W k ,W v ,W o );

[0133] Among them, MultiHead represents multi-head splicing, head1 represents the first attention head, and head h represents the hth attention head, W q ,W k ,W v ,W o is the attention mechanism parameter, where each attention head is expressed as:

[0134]

[0135] M = K = V = Q;

[0136] Among them, head c represents the cth attention head, Attention represents the self-attention mechanism, is the attention mechanism parameter;

[0137] In the present invention, the multi-head self-attention mechanism allows the features of each modality to perceive and pay attention to the features of other modalities. Through the self-attention mechanism, the features of each modality can extract relevant information from other modalities, thereby realizing cross-modal interaction and obtaining fused multi-modal features. This cross-modal information fusion has a synergistic effect on the overall humor recognition, thereby improving the accuracy of humor recognition.

[0138] Replace U″ in M′ t ,U″ v ,U″ a ,U″ f Combine and get the joint vector Ufinal , which is specifically expressed by the following formula:

[0139]

[0140] The joint vector U final Input the fully connected layer and use the Softmax function to obtain the predicted probability distribution P that represents the humor polarity, which is specifically expressed by the following formula:

[0141] P = SoftMax(U final W2+b2);

[0142] Among them, W2 and b2 are the parameters of the fully connected layer.

[0143] The predicted probability distribution P includes the probability of humor, the probability of non-humor, and the probability of neutrality.

[0144] The present invention also provides a GACA-based humor recognition method, which is specifically implemented through step B.

[0145] Step B: Attention network is used to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Update the updated features, and pass them through the cross-attention mechanism, linear transformation, multi-head cross-attention mechanism and multi-head self-attention layer to obtain the predicted probability distribution P that characterizes the humor polarity of the input video data; combine Figure 3 , specifically including the following steps:

[0146] Step B1: Use graph attention network to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Update to get the updated text modal feature f t , the updated humor modal feature f f , updated visual modality features f v and the updated audio modal features f a ;

[0147] Step B1 specifically includes:

[0148] The text modality features U t , visual modality features U v , audio modal features U a and humor modality feature U fPerform feature extraction and dependency analysis to obtain the node feature matrix of the text mode, the node feature matrix of the visual mode, the node feature matrix of the audio mode, the node feature matrix of the humor mode, and the adjacency matrix, where the adjacency matrix represents the grammatical dependency relationship between words;

[0149] Specifically, the node feature matrix obtains the features of each word through GloVe word embedding, and the adjacency matrix is ​​obtained by dependency analysis using SpyCy.

[0150] Among them, the node feature matrix of the text mode includes the text feature vector corresponding to each word, and each text feature vector is used as a node of the text graph to obtain an initial text graph; the node feature matrix of the visual mode includes the visual feature vector corresponding to each word, and each visual feature vector is used as a node of the visual graph to obtain an initial visual graph; the node feature matrix of the audio mode includes the audio feature vector corresponding to each word, and each audio feature vector is used as a node of the audio graph to obtain an initial audio graph; the node feature matrix of the humor mode includes the humor feature vector corresponding to each word, and each humor feature vector is used as a node of the humor graph to obtain an initial humor graph;

[0151] Initialize the weight W3 of the linear transformation through the uniform distribution formula. The uniform distribution formula is expressed as:

[0152]

[0153] Among them, n in is the number of nodes in the input layer, n out is the number of output layer nodes, gain is an optional scaling factor used to adjust the initialization standard deviation.

[0154] For each node in the initial text graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the text feature vector of the node and the text feature vector of an adjacent node are multiplied by the weight W3 of the initialized linear transformation, and the result of multiplying the text feature vector of the node and the result of multiplying the text feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node, which is specifically implemented by the following formula:

[0155] e mn =a([W3A m ||W3A n ]),n∈N m ;

[0156] Among them, e mn is the correlation coefficient between node m and node n, || represents the connection operation, a(·) represents the mapping operation, Am is the text feature vector corresponding to node m, A n is the text feature vector corresponding to node n, N m Represents the subscript set of nodes connected to node m;

[0157] The correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, which is specifically implemented by the following formula:

[0158]

[0159] Among them, α mn is the normalized attention coefficient of node m and node n, e mr is the correlation coefficient between node m and node r;

[0160] The normalized attention coefficients are weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated text modality feature f t , which is specifically achieved through the following formula:

[0161]

[0162] Among them, A′ m (l) represents the output vector of node m at the lth attention head, i.e., the updated text modality feature f t , L represents the total number of attention heads, σ(·) represents the activation function, represents the normalized attention coefficient of the lth attention head, represents the weight matrix of the lth attention head;

[0163] Similarly, for each node in the initial visual graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the visual feature vector of the node and the visual feature vector of an adjacent node are multiplied by the weight W3 of the initialized linear transformation respectively, and the result of multiplying the visual feature vector of the node and the result of multiplying the visual feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated visual modal feature f v ;

[0164] For each node in the initial audio graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the audio feature vector of the node and the audio feature vector of an adjacent node, and the result of multiplying the audio feature vector of the node and the result of multiplying the audio feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node, and the correlation coefficient between the node and the adjacent node is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated audio modal feature f a ;

[0165] For each node in the initial humor graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the weight W3 of the initialized linear transformation is respectively multiplied with the humor feature vector of the node and the humor feature vector of an adjacent node, and the result of multiplying the humor feature vector of the node and the result of multiplying the humor feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated humor modal feature f f .

[0166] Step B2: Use the cross-attention mechanism to update the text modality feature f t and the updated humor modal feature f f Perform modal fusion to obtain the first fusion feature f t,f , which is specifically expressed by the following formula:

[0167]

[0168] in, d t is the output dimension of the encoder corresponding to the updated text modality feature, d f The updated humor modality feature corresponds to the output dimension of the encoder;

[0169] Through the cross attention mechanism, the updated visual modality feature f v and the updated audio modal features f a Perform modal fusion to obtain the second fusion feature f a,v , which is specifically expressed by the following formula:

[0170]

[0171] in d v is the output dimension of the encoder corresponding to the updated visual modality feature, d a The output dimension of the encoder corresponding to the updated audio modality feature;

[0172] Step B3: For the first fusion feature f t,f and the second fusion feature f a,v Perform linear transformation to obtain the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is implemented by the following formula:

[0173]

[0174] in, and is the weight matrix, which can be learned, and K1 is the first fusion feature f t,f The key feature matrix, V1 is the first fusion feature f t,f The feature matrix of the value of, K2 is the second fusion feature f a,v The key feature matrix, V2 is the second fusion feature f a,v The feature matrix of the values ​​of ;

[0175] Step B4: Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, and the outputs of all attention heads are aggregated to obtain the first output O1 and the second output O2;

[0176] Wherein, step B4 specifically includes:

[0177] Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, which is implemented by the following formula:

[0178]

[0179] Among them, d k Indicates the dimension of the key, Attention h(Q1, K2, V2) is the output of the hth attention head after multi-head attention calculation of Q1, K2 and V2. h (Q2, K1, V1) is the output of the hth attention head after multi-head attention calculation of Q2, K1 and V1; H represents the total number of attention heads, is the index of the h-th attention head in Q1; is the index of the h-th attention head in Q2; is the index of the h-th attention head in K2; for The index of the h-th attention head in ; is the index of the h-th attention head in V1; is the index of the h-th attention head in V2; for The transpose of for The transpose of

[0180] Aggregate the outputs of all attention heads to get, which is implemented by the following formula:

[0181] O1=Concat([Attention 1 (Q1,K2,V2),...,Attention H (Q1,K2,V2)]);

[0182] O2=Concat([Attention 1 (Q2,K1,V1),...,Attention H (Q2,K1,V1)]);

[0183] Among them, H represents the total number of attention heads, O1 is the concatenated output of all attention heads after Q1 performs multi-head attention calculation with K2 and V2, and O2 is the concatenated output of all attention heads after multi-head attention calculation with K1 and V1.

[0184] Step B5: Combine O1 with the first fusion feature f t,f Connect and get the updated modal feature f′ t,f , O2 is combined with the second fusion feature f a,v Connect and get the updated modal feature f′ a,v , which is specifically expressed by the following formula:

[0185] f′ t,f =O1+f t,f ;

[0186] f′a,v =O2+f a,v ;

[0187] Step B6: f′ t,f and f′ a,v Input into the multi-head self-attention layer in sequence to obtain the processed information feature f a,v,f,t , the multi-head self-attention layer is used to update each modal feature of the input. A normalization and residual connection layer is added after each self-attention sublayer in the multi-head self-attention layer; the processed information features are input into the maximum pooling layer to obtain the final feature f final , the final feature f final Input the softmax layer to obtain the predicted probability distribution P that represents the humor polarity of the input video data, which is specifically expressed by the following formula:

[0188] P = softmax(f final W5+b5);

[0189] Among them, W5 and b5 are the parameters of the softmax layer.

[0190] Step 7: Use the predicted probability distribution P as the predicted value, and obtain the true value representing the humor polarity corresponding to the input video data. Calculate the loss function value based on the predicted value and the true value, which is specifically implemented by the following formula:

[0191]

[0192] Among them, BinaryCrossEntropy is the loss function value, N is the number of input video data, y i is the true value of the i-th input video data, y′ i is the predicted value of the i-th input video data;

[0193] Among them, the loss function used in the present invention is binary cross entropy loss.

[0194] Through the loss function value, the embedding vector matrix, deep convolutional neural network CNN, Transformer encoder, linear layer parameters W1 and b1, and fully connected layer parameters W2 and b2 are updated until the loss function value is less than the preset threshold. At this time, the output predicted probability distribution P is the humor polarity prediction result of the input video data.

[0195] Humor recognition for multimodal data aims to detect whether there is humor in the given data. As a special task in the field of natural language processing, it is very interesting but also faces severe challenges.

[0196] Two key points in the present invention are as follows:

[0197] A humor recognition method based on HTT is proposed. The present invention summarizes the currently recognized mainstream humor theory and uses it as the theoretical basis for obtaining humor external knowledge. Appropriate features are selected for encoding and representation to act as the injection of external knowledge. The present invention treats external knowledge as a separate modality and fuses it with text, audio, and video modalities. The key point of this invention is to solve the problem of external knowledge introduction and modality fusion scheme.

[0198] A humor recognition method based on GACA is proposed. The present invention proposes to use graph attention network for data enhancement to make up for the scarcity of current data sets. And the cross-attention mechanism is used to optimize modal fusion to create a more effective modal fusion solution.

[0199] Based on the above scheme, the present invention uses datasets with language, acoustic and visual modalities for experiments. Currently, the only open source datasets that meet the conditions are UR-FUNNY and MUStARD, which provide labeled data for natural language processing and machine learning model training, especially for identifying humorous contexts in videos or texts. In order to evaluate the performance of the model, the present invention evaluates it by binary classification accuracy and F1 score. Combined with Table 1, through comparative experiments, the HTT model achieved an accuracy improvement of 0.06% and 1.34% compared to the optimal baseline model. Combined with Table 2, the GACA model achieved an accuracy improvement of 0.77% and 1.98% compared to HTT, thereby verifying the effectiveness and advancement of the model of the present invention.

[0200] Table 1 Experimental results of HTT and baseline comparison

[0201]

[0202] Table 2 Experimental results of GACA compared with other baselines

[0203]

[0204]

[0205] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A multimodal humor recognition method, characterized in that: include: Step 1: Acquire a plurality of input video data, wherein the input video data includes input text, face image and audio information; Step 2: For each input video data in step 1, perform word segmentation and vector embedding on the input text to obtain the final text vector, and input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t ; Step 3: For each input video data in step 1, align and calibrate the face image by affine transformation technology to obtain an aligned face image; crop the face part in the aligned face image to obtain a first image, input the first image into a deep convolutional neural network CNN to obtain a face feature vector, perform L2 normalization on the face feature vector to obtain a normalized face feature vector, and select action units and face shape parameter features from the normalized face feature vector as the final face image feature f v ; Step 4: For the audio information in step 1, extract the pitch information from the audio information, extract the pitch information through COVAREP to obtain multiple pitch features, and process the multiple pitch features through SPS software to obtain the audio feature f after dimensionality reduction a ; Step 5: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and then extract the humorous features U according to the humorous features and the text modality features U t , the final face image feature f v and audio feature f a , get the visual modality feature U v , audio modal features U a and humor modality feature U f ; Step 6: Text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Processing is performed to obtain a predicted probability distribution P that represents humor polarity, which is specifically achieved through step A or step B: Step A: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Input the linear layer to obtain the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f , for the final text modality feature U′ t , the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U′ f The spliced ​​results are processed through a multi-head self-attention mechanism and a fully connected layer to obtain a predicted probability distribution P that represents the humor polarity of the input video data; Step B: Attention network is used to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Update, and pass the updated features through the cross-attention mechanism, linear transformation, multi-head cross-attention mechanism and multi-head self-attention layer to obtain the predicted probability distribution P that characterizes the humor polarity of the input video data; Step 7: Use the predicted probability distribution P as the predicted value, and obtain the true value representing the humor polarity corresponding to the input video data. Calculate the loss function value based on the predicted value and the true value, which is specifically implemented by the following formula: Among them, BinaryCrossEntropy is the loss function value, N is the number of input video data, y i is the true value of the i-th input video data, y′ i is the predicted value of the i-th input video data; Through the loss function value, the embedding vector matrix, deep convolutional neural network CNN, Transformer encoder, linear layer parameters W1 and b1, and fully connected layer parameters W2 and b2 are updated until the loss function value is less than the preset threshold. At this time, the output predicted probability distribution P is the humor polarity prediction result of the input video data.

2. A multimodal humor recognition method according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: For each input video data in step 1, the input text is segmented by the WordPiece word segmenter of the ALBERT model to obtain the segmented text S′={(t1, t2, ..., t j ),(p1,p2,...,p k )}, where (t1, t2, ..., t j ) is the context of the input text, t1 represents the first word in the context, t2 represents the second word in the context, and t j represents the jth word in the context, j represents the total number of subwords in the context, (p1, p2, ..., p k ) is the punchline of the input text, p1 represents the first word of the punchline, p2 represents the second word of the punchline, and p k The kth word representing the punchline, k is the total number of subwords of the punchline, and tags are added to the segmented text. Specifically, a [CLS] tag is added to the beginning of the segmented text, a [SEP] tag is added between the context and the punchline, and a [SEP] tag is added to the end of the segmented text to obtain a text sequence X = [CLS]t1...t j [SEP]p1...p k [SEP]; Step 2.2: Convert the text sequence into an embedded vector and obtain the embedded vector matrix X1 = E [CLS] E1...E j E [SEP] E′1...E′ k E [SEP] , where E [CLS] is the embedding vector of [CLS], E1 is the embedding vector of t1, E j t j The embedding vector, E [SEP] is the embedding vector of [SEP], E′1 is the embedding vector of p1, and E′ k For p k The embedding vector, E [SEP] is the embedding vector of [SEP]; for the context and jokes in the embedding vector matrix, different segment embeddings are added to obtain the first embedding representation, and position embedding is added to each word in the first embedding representation to obtain the final text vector embedding representation, where the position embedding represents the position of the word in the first vector embedding representation; Step 2.3: Input the final text vector embedding representation into the ALBERT model to obtain the text modality feature U t .

3. A multimodal humor recognition method according to claim 2, characterized in that: Step 5 specifically includes: Step 5.1: For each input video data in step 1, extract the humorous features of the context and the humorous features of the jokes from the input text, and convert the humorous features of the context and the humorous features of the jokes to obtain the converted humorous features X f =[PAD]C f [SEP]P f , where C f is the humorous feature corresponding to the context, P f is the humor feature corresponding to the punchline, [PAD] is a placeholder, and [SEP] is used to convert C f With P f separate; Step 5.2: Use the P2FA forced alignment model to extract the time corresponding to each word in the segmented text described in step 2.1, and align the facial image feature f according to the time corresponding to each word. v and audio feature f a Segmentation is performed, and the segmented face image features are represented as X v =[PAD]C v [SEP]P v , C v is the facial image feature corresponding to the context, P v is the facial image feature corresponding to the laugh point, and the segmented audio feature is represented as X a =[PAD]C a [SEP]P a , C a is the audio feature corresponding to the context, P a The audio features corresponding to the laugh points; Step 5.3: The segmented face image feature X v Input Transformer encoder to get visual modality feature U v , which is specifically expressed by the following formula: U v =TransformerEncoder(X v ); The segmented audio feature X a Input Transformer encoder to get audio modality feature U a , which is specifically expressed by the following formula: U a =TransformerEncoder(X a ); Add humor feature X f Input Transformer encoder to get humor modality feature U f , which is specifically expressed by the following formula: U f =TransformerEncoder(X f )。 4. The multimodal humor recognition method according to claim 1, characterized in that: Step A specifically includes: Step A1: Transform the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f , respectively input into the linear layer to obtain the final text modality feature U of the same dimension t ′, the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U f ′, the formula of the linear layer is expressed as: Y=XW1+b1; Among them, W1 is the learning weight matrix, b1 is the bias vector; Specifically, the text modality feature U t , visual modality features U v , audio modal features U a and humor modality features into X to obtain the final text modality feature U t ′, the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U f ′; Step A2: Final text modality feature U t ′, the final visual modality feature U′ v , the final audio modal feature U′ a and the final humor modal feature U f ' to concatenate, and obtain the matrix M = [U t ′,U′ v ,U′ a ,U f ′], through the multi-head self-attention mechanism, the matrix M is processed to obtain M′=[U t ″,U′ v ′,U′ a ′,U f ″], which is specifically achieved through the following formula: M′M=MultiHead(M,θ)=Concat(head1,...,head h )W o ; θ={W q ,W k ,W v ,W o ); Among them, MultiHead represents multi-head splicing, head1 represents the first attention head, and head h represents the hth attention head, W q ,W k ,W v ,W o is the attention mechanism parameter, where each attention head is expressed as: M = K = V = Q; Among them, head c represents the cth attention head, Attention represents the self-attention mechanism, is the attention mechanism parameter; Put U in M' t ″,U′ v ′,U′ a ′,U f ″ to obtain the joint vector U final , which is specifically expressed by the following formula: The joint vector U final Input the fully connected layer and use the Softmax function to obtain the predicted probability distribution P that represents the humor polarity, which is specifically expressed by the following formula: P=SoftMax(U final W2+b2); Among them, W2 and b2 are the parameters of the fully connected layer.

5. The multimodal humor recognition method according to claim 1, characterized in that: Step B specifically includes: Step B1: Use graph attention network to analyze the text modality feature U t , visual modality features U v , audio modal features U a and humor modality feature U f Update to get the updated text modal feature f t , updated humor modality feature f f , updated visual modality feature f v and the updated audio modal features f a ; Step B2: Use the cross-attention mechanism to update the text modality feature f t and the updated humor modal feature f f Perform modal fusion to obtain the first fusion feature f t,f , which is specifically expressed by the following formula: Through the cross attention mechanism, the updated visual modality feature f v and the updated audio modal features f a Perform modal fusion to obtain the second fusion feature f a,v , which is specifically expressed by the following formula: Step B3: For the first fusion feature f t,f and the second fusion feature f a,v Perform linear transformation to obtain the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is implemented by the following formula: in, and is the weight matrix, K1 is the first fusion feature f t,f The key feature matrix, V1 is the first fusion feature f t,f The feature matrix of the value of, K2 is the second fusion feature f a,v The key feature matrix, V2 is the second fusion feature f a,v The feature matrix of the values ​​of ; Step B4: Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, and the outputs of all attention heads are aggregated to obtain the first output O1 and the second output O2; Step B5: Combine O1 with the first fusion feature f t,f Connect and get the updated modal feature f′ t,f , O2 is combined with the second fusion feature f a,v Connect and get the updated modal feature f′ a,v , which is specifically expressed by the following formula: f′ t,f =O1+f t,f ; f′ a,v =O2+f a,v ; Step B6: f′ t,f and f′ a,v Input into the multi-head self-attention layer in sequence to obtain the processed information feature f a,v,f,t , input the processed information features into the maximum pooling layer to obtain the final feature f final , the final feature f final Input the softmax layer to obtain the predicted probability distribution P that represents the humor polarity of the input video data, which is specifically expressed by the following formula: P=softmax(f final W5+b5); Among them, W5 and b5 are the parameters of the softmax layer.

6. A multimodal humor recognition method according to claim 5, characterized in that: Step B1 specifically includes: The text modality features U t , visual modality features U v , audio modal features U a and humor modality feature U f Perform feature extraction and dependency analysis to obtain the node feature matrix of the text mode, the node feature matrix of the visual mode, the node feature matrix of the audio mode, the node feature matrix of the humor mode, and the adjacency matrix, where the adjacency matrix represents the grammatical dependency relationship between words; Among them, the node feature matrix of the text mode includes the text feature vector corresponding to each word, and each text feature vector is used as a node of the text graph to obtain an initial text graph; the node feature matrix of the visual mode includes the visual feature vector corresponding to each word, and each visual feature vector is used as a node of the visual graph to obtain an initial visual graph; the node feature matrix of the audio mode includes the audio feature vector corresponding to each word, and each audio feature vector is used as a node of the audio graph to obtain an initial audio graph; the node feature matrix of the humor mode includes the humor feature vector corresponding to each word, and each humor feature vector is used as a node of the humor graph to obtain an initial humor graph; The weight W3 of the linear transformation is initialized through the uniform distribution formula. For each node in the initial text graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix. The weight W3 of the linear transformation is initialized and multiplied with the text feature vector of the node and the text feature vector of an adjacent node respectively. The result of multiplying the text feature vector of the node and the result of multiplying the text feature vector of the adjacent node are concatenated, and the concatenated result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node. This is specifically implemented by the following formula: <h2 style=";text-align:left;direction:ltr">e<h2 style=";text-align:left;direction:ltr"> mn <h2 style=";text-align:left;direction:ltr"> (a([W3A<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> ||W3A<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ]),n∈N<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> ; Among them, e mn is the correlation coefficient between node m and node n, || represents the connection operation, a(·) represents the mapping operation, and A m is the text feature vector corresponding to node m, A n is the text feature vector corresponding to node n, N m Represents the subscript set of nodes connected to node m; The correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, which is specifically implemented by the following formula: Among them, α mn is the normalized attention coefficient of node m and node n, e mr is the correlation coefficient between node m and node r; The normalized attention coefficients are weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated text modality feature f t , which is specifically achieved through the following formula: Among them, A′ m (l) represents the output vector of node m at the lth attention head, i.e., the updated text modality feature f t , L represents the total number of attention heads, σ(·) represents the activation function, represents the normalized attention coefficient of the lth attention head, represents the weight matrix of the lth attention head; Similarly, for each node in the initial visual graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the visual feature vector of the node and the visual feature vector of an adjacent node are multiplied by the weight W3 of the initialized linear transformation respectively, and the result of multiplying the visual feature vector of the node and the result of multiplying the visual feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated visual modal feature f v ; For each node in the initial audio graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the initialized linear transformation weight W3 is respectively multiplied with the audio feature vector of the node and the audio feature vector of an adjacent node, and the result of multiplication with the audio feature vector of the node and the result of multiplication with the audio feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent node, and the correlation coefficient between the node and the adjacent node is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated audio modal feature f a ; For each node in the initial humor graph, the adjacent nodes adjacent to the node are determined according to the adjacency matrix, and the weight W3 of the initialized linear transformation is respectively multiplied with the humor feature vector of the node and the humor feature vector of an adjacent node, and the result of multiplying the humor feature vector of the node and the result of multiplying the humor feature vector of the adjacent node are spliced, and the spliced ​​result is mapped through a layer of feedforward neural network to obtain the correlation coefficient between the node and the adjacent nodes, and the correlation coefficient between the node and the adjacent nodes is normalized to obtain the normalized attention coefficient, and the normalized attention coefficient is weighted summed, and the result of the weighted summation is passed through the graph attention network of the multi-head attention mechanism to obtain the output of the node after the graph attention network, that is, the updated humor modal feature f f .

7. The multimodal humor recognition method according to claim 5, characterized in that: Step B4 specifically includes: Through the multi-head cross attention mechanism, the first fusion feature f t,f The query vector Q1 and the second fusion feature f a,v The query vector Q2 is processed to obtain the output of each attention head, which is implemented by the following formula: Among them, d k Indicates the dimension of the key, Attention h (Q1, K2, V2) is the output of the hth attention head after multi-head attention calculation of Q1, K2 and V2. h (Q2, K1, V1) is the output of the hth attention head after multi-head attention calculation of Q2, K1 and V1; H represents the total number of attention heads, is the index of the h-th attention head in Q1; is the index of the h-th attention head in Q2; is the index of the h-th attention head in K2; for The index of the h-th attention head in ; is the index of the h-th attention head in V1; is the index of the h-th attention head in V2; for The transpose of for The transpose of Aggregate the outputs of all attention heads to get, which is implemented by the following formula: O1=Concat([Attention 1 (Q1,K2,V2),...,Attention H (Q1,K2,V2)]); O2=Concat([Attention 1 (Q2,K1,V1),...,Attention H (Q2,K1,V1)]); Among them, H represents the total number of attention heads, O1 is the concatenated output of all attention heads after Q1 performs multi-head attention calculation with K2 and V2, and O2 is the concatenated output of all attention heads after multi-head attention calculation with K1 and V1.

Images