Residual fusion language feature-based multi-modal Chinese siphonage identification method

Abstract

The invention relates to a multi-modal Chinese interference identification method based on residual fusion language features, which comprises the following steps of: (1) extracting words with anti-interference and non-anti-interference meanings from a self-constructed image-text multi-modal Chinese interference identification data set by utilizing a chi-square statistical method, and constructing a language feature system; (2) extracting language features of words in the language feature system by using a TextCNN model; (3) extracting image features by using a ResNet34 model; (4) introducing a cross attention mechanism, and establishing association between image-text features; and (5) carrying out feature fusion to obtain a classification result. The limitation of the existing method can be improved, and the accuracy of the phage recognition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal Chinese irony recognition, and in particular to a multimodal Chinese irony recognition method using residual fusion language features. Background Art

[0002] At present, for the purpose of accurately identifying the ironic emotions of multimodal reviews, the existing technology generally integrates text, emoticons and image emotional features, builds a tourism review irony recognition model based on BiLSTM, emoji2vec and FCNN models, and after vectorizing the three features, performs bimodal and trimodal feature vector fusion respectively, and finally uses Sigmoid and Softmax classifiers to classify the fused vectors and output the tourism review irony results. According to the above design route, the multimodal fusion tourism review irony recognition model can be divided into input layer, feature representation layer, feature fusion layer, and irony recognition layer. However, in the feature fusion layer, only simple weighting of modal features is used, and the information interaction between modalities is not considered.

[0003] Through the study of existing multimodal Chinese irony recognition, it is found that the overall design of the model is mostly focused on the training of deep learning models, and the relevant models are designed with the focus on information fusion and interaction between modalities and capturing "inter-modal contradictions", while ignoring the explicit language characteristics of the ironic sentences themselves. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal Chinese irony recognition method using residual fusion language features to improve the limitations of the existing methods and the accuracy of irony recognition in view of the shortcomings of the existing technologies.

[0005] The technical problem to be solved by the present invention is achieved by the following technical solution. The present invention is a multimodal Chinese irony recognition method based on residual fusion language features, and the steps of the method are as follows:

[0006] (1) Using the chi-square statistical method, we extracted words with ironic and non-ironic meanings from a self-constructed multimodal Chinese irony recognition dataset and constructed a language feature system.

[0007] (2) Use the TextCNN model to extract the language features of words in the language feature system;

[0008] (3) Use the ResNet34 model to extract image features;

[0009] (4) Introduce the cross-attention mechanism to establish the association between image and text features;

[0010] (5) Feature fusion to obtain classification results.

[0011] The technical problem to be solved by the present invention can be further achieved by the following technical solution. For the multimodal Chinese irony recognition method of residual fusion language features described above, the specific operation of step (2) is:

[0012] Using Word2ve c The model learns the semantic information of netizens' comments on emergencies from massive texts to obtain text vector representation as the input of the text model;

[0013] The set of netizens’ text comments in the multimodal irony recognition dataset of emergencies is X = {x1, x2, …, x i ,…,x n}, x i Represents a text comment, each comment x obtained by word segmentation i =(x i1 ,…,x ii ,…,x ik ), x ik Represents the kth word of the i-th text comment, each word trained by Word2vec Represents the m-dimensional word vector of the k-th word;

[0014] The input of the TextCNN model is an n×d matrix X, where each row represents the d-dimensional word vector representation of a word in a text;

[0015] For the matrix convolution operation, a sliding window calculation is performed on the matrix to obtain the feature map matrix C. The convolution calculation is as follows:

[0016] C i =f(W×X i：i+h-1 +b)

[0017] Among them, X i：i+h-1 represents the i-th row to the i+h-1th column of the matrix X, W is the convolution kernel, its size is h×d, b is the bias vector, and f is the activation function;

[0018] For each feature map C obtained, the pooling vector t is obtained through the maximum pooling operation:

[0019] t=max(C)

[0020] All pooled vectors t are concatenated to form a long vector T, which is then passed through a fully connected layer for classification or for subsequent applications;

[0021] The language feature vector L is obtained in the same way for subsequent use.

[0022] The technical problem to be solved by the present invention can be further achieved by the following technical solution. For the multimodal Chinese irony recognition method of residual fusion language features described above, the specific operation of step (3) is:

[0023] The image set P corresponding to the netizens’ text comments in the multimodal irony recognition dataset of emergencies is {p1, p2, …, p i ,…,p n}, the initial size of each image pi in P is 224×224×3. After the image passes through several convolutional layers to the last convolutional layer (conv 5), the image size becomes 7×7×512. After the average pooling layer (avg pool), the output size becomes 1×1×512, and the pooled image feature i is obtained. The image feature i is input into the fully connected layer to obtain the image feature I:

[0024] I=Linear(i)

[0025] Among them, Linear() represents a fully connected layer.

[0026] The technical problem to be solved by the present invention can be further achieved by the following technical solution. For the multimodal Chinese irony recognition method of residual fusion language features described above, the specific operation of step (4) is:

[0027] The text features T and image features I obtained above are used to learn the interactive information between modalities through the cross-attention mechanism and establish the connection between them. The cross-attention mechanism is applied to the modal fusion process by using the residual connection method, and a feature fusion module is designed:

[0028] The text feature is used as the query Q, the image feature is used as the key K and the value V, and the attention feature at the text level is obtained by applying the calculation method of the attention mechanism. The calculation formula is as follows:

[0029]

[0030] Among them, W Q , W K , W V It is the parameter matrix that can be learned during model training. k The purpose is to balance the impact of different dimensions on similarity calculation;

[0031] Connect the outputs of multiple heads to get the total text-level attention feature. In this article, set n_heads = 0. Then, connect the original text feature T and the text-level attention feature Att(T,I) obtained by cross-attention by using residual connection:

[0032] AttT=Att(T,I)+T

[0033] At this point, the total output AttT after the text layer attention features is obtained;

[0034] Similarly, the image feature is used as the query Q, the text feature is used as the key K and the value V. The calculation method is the same as that of calculating the text layer attention score. The image layer attention score is calculated by guiding the text information through the image, and the total output AttI after the image layer attention feature is obtained;

[0035] The obtained text layer attention feature AttT and image layer attention feature AttI are connected to form the image-text output AttTI of the image-text modality cross attention fusion module:

[0036] AttTI=cat(AttT,AttI)

[0037] The acquired language feature L and the image and text output AttTI are connected by a simple connection to obtain the final feature representation AttTIL of modal fusion:

[0038] AttTIL=cat(L,AttTI).

[0039] The technical problem to be solved by the present invention can be further achieved by the following technical solution. For the multimodal Chinese irony recognition method of residual fusion language features described above, the test method of the method is:

[0040] (1) Use a multimodal Chinese irony recognition dataset built on the Weibo platform to test the effect;

[0041] (2) The text and the image are sent to the convolutional neural network for forward propagation. During the forward propagation, the convolutional layer is used for feature extraction, and then the data is mapped through the normalization layer. Finally, the activation function is used for nonlinear mapping to obtain the features of the text and the image respectively. The features of the obtained text and image are processed with cross attention and residual connection to obtain the overall irony recognition features.

[0042] (3) Obtained irony recognition features and label calculation loss;

[0043] (4) After obtaining the loss function, reverse gradient calculation is performed;

[0044] (5) Reverse weight update is performed in combination with the learning rate. The learning rate is initially set according to international regional standards. As the number of training iterations increases, the learning rate decreases. Each time the weight is updated, it means that a training session is completed.

[0045] (6) Repeat the above steps and continue to input synthetic data to train the network until the loss value tends to be stable and the network converges;

[0046] (7) The trained parameters are saved. During the test phase, the saved model is directly loaded, and then forward propagation is performed on the image and text to be tested. Backward propagation is no longer performed at this time because there is no need to update or save the weights.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. Improve the accuracy of irony recognition

[0049] By comprehensively considering multiple modal data, the present invention can capture richer and more comprehensive information, thereby improving the accuracy and robustness of irony recognition;

[0050] 2. Improve the limitations of existing methods

[0051] Aiming at the limitations of existing irony recognition methods in multimodal fusion, which usually focus on building auxiliary modalities for pictures but ignore the characteristics of Chinese ironic sentences themselves, the present invention proposes a multimodal irony recognition method that integrates language features from a new perspective to effectively overcome these limitations. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A specific flow chart constructed for the language feature system of the present invention;

[0053] Figure 2 It is a flow chart for building the overall model of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] A multimodal Chinese irony recognition method based on residual fusion language features, the steps are as follows:

[0056] Step 1: Use the chi-square statistical method to extract words with ironic and non-ironic meanings from the self-constructed multimodal Chinese irony recognition dataset and construct a language feature system;

[0057] Conditions: A multimodal Chinese irony recognition dataset containing ironic and non-ironic comments has been prepared;

[0058] Function: Filter out keywords with ironic and non-ironic meanings and establish a language feature system.

[0059] Specific process of construction Figure 1 As shown, through the above process, the following ironic and non-ironic vocabulary related results are obtained, and some of the results are shown in the following three tables;

[0060] Table 1 Chi-square statistics of some ironic vocabulary collocation rules

[0061] serial number Some results of collocation rules Chi-square value 1 Before...After 1530.0006 2 Listen to me...thank you 1528.0026 3 As expected... 405.1525 4 Well... 12.5440 5 Again… 2.7910

[0062] Table 2 Chi-square statistics of some ironic words

[0063]

[0064]

[0065] Table 3 Chi-square statistics of non-ironic vocabulary

[0066]

[0067] Step 2: Use the TextCNN model to extract the language features of words in the language feature system;

[0068] The Word2vec model is used to learn the semantic information of netizens’ comments on emergencies in massive texts to obtain text vector representation as the input of the text model; the netizens’ text comment set X in the multimodal irony recognition dataset of emergencies is i , ..., x n}, x i Represents a text comment; each comment x obtained by word segmentation i =(x i1 , ..., x ii , ..., x ik ), x ik Represents the kth word of the i-th text comment; each word trained by Word2vec Represents the m-dimensional word vector of the k-th word.

[0069] The input of the TextCNN model is an n×d matrix X, where each row represents the d-dimensional word vector representation of a word in a text.

[0070] For the matrix convolution operation, a sliding window calculation is performed on the matrix to obtain the feature map matrix C. The convolution calculation is as follows:

[0071] C i=f(W×X i：i+h-1 +a)

[0072] Among them, X i：i+h-1 represents the i-th row to the i+h-1-th column of the matrix X, W is the convolution kernel, whose size is h×d, b is the bias vector, and f is the activation function.

[0073] For each feature map C obtained, the pooling vector t is obtained through the maximum pooling operation:

[0074] t=max(C)

[0075] All pooled vectors t are concatenated to form a long vector T, which is then passed through a fully connected layer for classification or for subsequent applications.

[0076] The language feature vector L is obtained in the same way for subsequent use.

[0077] Condition: The trained TextCNN model and the dataset required for feature extraction are ready.

[0078] Purpose: Use deep learning models to extract language features of text data for subsequent irony recognition tasks.

[0079] Step 3: Use the ResNet34 model to extract image features;

[0080] The image set P corresponding to the netizens’ text comments in the multimodal irony recognition dataset of emergencies is {p1, p2, ..., p i , ..., p n}, each image p in P i The initial size of is 224×224×3. After the image passes through several convolutional layers to the last convolutional layer (conv 5), the image size becomes 7×7×512. After the average pooling layer (avg pool), the output size becomes 1×1×512, and the pooled image feature i is obtained. The image feature i is input into the fully connected layer to obtain the image feature I:

[0081] I=Linear(i)

[0082] Among them, Linear() represents a fully connected layer.

[0083] Condition: The trained ResNet model and the image dataset required for feature extraction are ready.

[0084] Function: Use deep learning models to extract features of image data and provide image information for multimodal fusion.

[0085] Step 4: Introduce the cross-attention mechanism to establish the association between image and text features;

[0086] The text features T and image features I obtained above are used to learn the interactive information between modalities through the cross-attention mechanism and establish the connection between them. To achieve this goal, the cross-attention mechanism is applied to the modality fusion process by using residual connections, and finally a feature fusion module is designed.

[0087] The text feature is used as the query Q, the image feature is used as the key K and the value V, and the attention feature at the text level is obtained by applying the calculation method of the attention mechanism. The calculation formula is as follows:

[0088]

[0089] Among them, W Q , W K , W V It is the parameter matrix that can be learned during model training. k The purpose is to balance the impact of different dimensions on similarity calculation.

[0090] By connecting the outputs of multiple heads, we can get the total text-level attention features. In this paper, n_heads=0 is set. Then, the original text feature T and the text-level attention feature Att(T, I) obtained by cross-attention are connected by using residual connection. This design not only retains the information of the original text features, but also obtains the interactive information of the image and text:

[0091] AttT=Att(T,I)+T

[0092] At this point, the total output AttT after the text layer attention features is obtained.

[0093] Similarly, the image feature is used as the query Q, the text feature is used as the key K and the value V, and the calculation method is the same as that of the text layer attention score. By guiding the text information with the image, the image layer attention score is calculated, and the total output AttI after the image layer attention feature is obtained. With this design, the model can fully pay attention to the interactive information between the image and the text, and better capture the inconsistent information between them.

[0094] The obtained text layer attention feature AttT and image layer attention feature AttI are connected to form the image-text output AttTI of the image-text modality cross attention fusion module:

[0095] AttTI=cat(AttT,AttI)

[0096] The language feature L obtained in the previous chapter and the image and text output AttTI are connected by a simple connection to obtain the final feature representation AttTIL of modal fusion:

[0097] AttTIL=cat(L,AttTI)

[0098] Condition: The algorithm and model of the cross-attention mechanism have been designed;

[0099] Function: Enhance the correlation between image and text features and improve the accuracy of irony recognition.

[0100] Step 5: Feature fusion to obtain classification results.

[0101] The present invention proposes a related testing method:

[0102] (1) The effect is tested using a multimodal Chinese irony recognition dataset built on the Weibo platform.

[0103] (2) The text and image are sent to the convolutional neural network for forward propagation. During the forward propagation, the convolutional layer is used for feature extraction, and then the data is mapped through the normalization layer. Finally, the activation function is used for nonlinear mapping to obtain the features of the text and image respectively. The features of the obtained text and image are processed with cross attention and residual connection to obtain the overall irony recognition features.

[0104] (3) Obtain the irony recognition features and labels (Label) to calculate the loss.

[0105] (4) After obtaining the loss function, reverse gradient calculation is performed.

[0106] (5) Reverse weight update is performed in combination with the learning rate. The learning rate is initially set according to international regional standards. As the number of training iterations increases, the learning rate continues to decrease. Each time the weight is updated, it means that a training session is completed.

[0107] (6) Repeat the above steps and continue to input synthetic data to train the network until the loss value tends to be stable and the network converges.

[0108] (7) The trained parameters are saved. During the test phase, the saved model is directly loaded, and then forward propagation is performed on the image and text to be tested. Backward propagation is no longer performed at this time because there is no need to update or save the weights.

[0109] The present invention implements parameter settings:

[0110] The model is implemented based on the PyTorch framework. The word vector dimension of the text data is set to 300 dimensions. Four different convolution kernel sizes are used in the text data feature extraction model TextCNN, which are 2, 3, 4 and 5, and the number of each size is 400. Adam is used in the model optimizer selection, and the model learning rate is set to 5e-4. For efficient training, the batch size is set to 32, and 100 epochs are trained. If the accuracy of the network does not improve after 50 rounds, the training process will be terminated early. In order to limit the length, the maximum length of text and language features is limited to 64.

[0111] The detailed parameter settings are shown in the following table;

[0112] Hyperparameters value Word vector dimension 300 Learning rate 5e-4 Batch size 32 Dropout rate 0.2 Epochs 100 Text length 64 Language feature length 64 Optimizer Adam Activation function Tanh

[0113] The overall model construction flow chart is shown in Figure 2 As shown;

[0114] By implementing the above steps, the specific technical solution can be effectively implemented, and the effectiveness and practicality of the technical solution in the irony identification task can be verified.

[0115] The present invention has the following advantages:

[0116] 1. Improve recognition accuracy

[0117] By fusing multimodal data and language features, the accuracy (Accuracy) of the present invention on a self-constructed data set reaches 91.32%, the precision (Precision) reaches 94.12%, the recall rate (Recall) reaches 86.70%, and the F1 score (F1 Score) reaches 90.26%, which can effectively improve the accuracy of irony recognition; compared with the analysis method of single modal data, multimodal fusion can capture the characteristics of irony more comprehensively, thereby improving the accuracy of recognition.

[0118] The following table shows the results of the comparison between this model and other models. LFMIT is the abbreviation of the model of the present invention.

[0119] The baseline models are divided into three categories for comparative experiments with the model proposed in this invention, including text model, image model and text-image multimodal model;

[0120] Text Model

[0121] TextCNN: This is the text feature extraction stage of the present invention, and the acquired features are used for text classification.

[0122] Bi-LSTM: Use the Bi-LSTM network to capture text features and input them into the classification layer for text classification.

[0123] BERT: It is a pre-trained language model based on the Transformer architecture, which has achieved good results in many NLP tasks. This paper applies it to the text classification task as a comparative experimental object.

[0124] Image Model

[0125] ResNet: Use the image vector after the ResNet fully connected layer as the image feature and classify it as a feature.

[0126] VGGNet: Use VGG network to extract image features and then perform irony prediction.

[0127] Multimodal model of images and texts

[0128] TextCNN+ResNet: This is the feature extraction part of the model of the present invention, which simply connects text features and image features as the total features for classification.

[0129] Bi-LSTM+ResNet: Similarly, Bi-LSTM is used to extract text features, and the other parts remain unchanged.

[0130] Bi-LSTM+CNN: Use Bi-LSTM to extract text features, and CNN to extract image features, then perform multimodal fusion and classification.

[0131] BERT+ResNet: Similarly, BERT is used as the feature extraction part of the text, and the rest remains unchanged.

[0132] LFMIT: A proposed Chinese multimodal irony recognition model integrating linguistic features.

[0133] LFMIBi: Bi-LSTM is used to replace the text feature extraction part, and the other parts remain unchanged.

[0134] LFMIT+Att: The attention mechanism is added before model fusion, and the other parts remain unchanged.

[0135]

[0136] 2. Explore novel perspectives

[0137] The present invention starts from the perspective of language features and focuses more on the feature analysis of Chinese ironic sentences themselves. This new perspective is helpful to explore new methods and technologies in the field of irony recognition.

Claims

1. A multimodal Chinese irony recognition method based on residual fusion language features, characterized by: The steps of this method are as follows: (1) Using the chi-square statistical method, we extracted words with ironic and non-ironic meanings from a self-constructed multimodal Chinese irony recognition dataset and constructed a language feature system. (2) Use the TextCNN model to extract the language features of words in the language feature system; (3) Use the ResNet34 model to extract image features; (4) Introduce the cross-attention mechanism to establish the association between image and text features; (5) Feature fusion to obtain classification results.

2. The multimodal Chinese irony recognition method based on residual fusion language features according to claim 1 is characterized by: The specific operation of step (2) is: The Word2vec model is used to learn the semantic information of netizens’ comments on emergencies from massive texts to obtain text vector representation as the input of the text model; The set of netizens’ text comments in the multimodal irony recognition dataset of emergencies is X = {x1, x2, ..., x i , ..., x n }, x i Represents a text comment, each comment x obtained by word segmentation i =(x i1 , ..., x ii , ..., x ik ), x ik Represents the kth word of the i-th text comment, each word trained by Word2vec Represents the m-dimensional word vector of the k-th word; The input of the TextCNN model is an n×d matrix X, where each row represents the d-dimensional word vector representation of a word in a text; For the matrix convolution operation, a sliding window calculation is performed on the matrix to obtain the feature map matrix C. The convolution calculation is as follows: C i =f(W×X i:i+h-1 +b) Among them, X i：i+h-1 represents the i-th row to the i+h-1th column of the matrix X, W is the convolution kernel, its size is h×d, b is the bias vector, and f is the activation function; For each feature map C obtained, the pooling vector t is obtained through the maximum pooling operation: t=max(C) All pooled vectors t are concatenated to form a long vector T, which is then passed through a fully connected layer for classification or for subsequent applications; The language feature vector L is obtained in the same way for subsequent use.

3. The multimodal Chinese irony recognition method based on residual fusion language features according to claim 1 is characterized by: The specific operation of step (3) is: The image set P corresponding to the netizens’ text comments in the multimodal irony recognition dataset of emergencies is {p1, p2, ..., p i , ..., p n }, the initial size of each image pi in P is 224×224×3. After the image passes through several convolutional layers to the last convolutional layer, the image size becomes 7×7×512. After the average pooling layer, the output size becomes 1×1×512, and the pooled image feature i is obtained. The image feature i is input into the fully connected layer to obtain the image feature I: I=Linear(i) Among them, Linear() represents a fully connected layer.

4. The multimodal Chinese irony recognition method based on residual fusion language features according to claim 1 is characterized by: The specific operation of step (4) is: The text features T and image features I obtained above are used to learn the interactive information between modalities through the cross-attention mechanism and establish the connection between them. The cross-attention mechanism is applied to the modal fusion process by using the residual connection method, and a feature fusion module is designed: The text feature is used as the query Q, the image feature is used as the key K and the value V, and the attention feature at the text level is obtained by applying the calculation method of the attention mechanism. The calculation formula is as follows: Among them, W Q , W K , W V It is a parameter matrix that can be learned during model training. The purpose of introducing dk is to balance the impact of different dimensions on similarity calculation; Connect the outputs of multiple heads to get the overall text-level attention features. In this paper, we set n _ heads = 0. Then, the original text feature T and the text-level attention feature Att(T, I) obtained by cross-attention are connected by using residual connection: AttT=Att(T,I)+T At this point, the total output AttT after the text layer attention features is obtained; Similarly, the image feature is used as the query Q, the text feature is used as the key K and the value V. The calculation method is the same as that of calculating the text layer attention score. The image layer attention score is calculated by guiding the text information through the image, and the total output AttI after the image layer attention feature is obtained; The obtained text layer attention feature AttT and image layer attention feature AttI are connected to form the image-text output AttTI of the image-text modality cross-attention fusion module: AttTI=cat(AttT,AttI) The acquired language feature L and the image and text output AttTI are connected by a simple connection to obtain the final feature representation AttTIL of modal fusion: AttTIL=cat(L,AttTTI).

5. The multimodal Chinese irony recognition method based on residual fusion language features according to claim 1 is characterized by: The test method for this method is: (1) Use a multimodal Chinese irony recognition dataset built on the Weibo platform to test the effect; (2) The text and the image are sent to the convolutional neural network for forward propagation. During the forward propagation, the convolutional layer is used for feature extraction, and then the data is mapped through the normalization layer. Finally, the activation function is used for nonlinear mapping to obtain the features of the text and the image respectively. The features of the obtained text and image are processed with cross attention and residual connection to obtain the overall irony recognition features. (3) Obtained irony recognition features and label calculation loss; (4) After obtaining the loss function, reverse gradient calculation is performed; (5) Reverse weight update is performed in combination with the learning rate. The learning rate is initially set according to international regional standards. As the number of training iterations increases, the learning rate decreases. Each time the weight is updated, it means that a training session is completed. (6) Repeat the above steps and continue to input synthetic data to train the network until the loss value tends to be stable and the network converges; (7) The trained parameters are saved. During the test phase, the saved model is directly loaded, and then forward propagation is performed on the image and text to be tested. Backward propagation is no longer performed at this time because there is no need to update or save the weights.

Images