Method and system for detecting irony in dialogue scene

Abstract

The invention provides an irony detection method and system in a dialogue scene, and the method comprises the steps: carrying out the feature extraction of multi-modal data in a dialogue, obtaining the features of each modal data, and enabling the multi-modal data to comprise a plurality of types of texts, videos and voices; fusing the features of each modal data with the features of other modal data by using a multi-head cross attention mechanism to obtain fused features, and processing the fused features of each modal data and other modal data by sequentially using an activation function and a normalized exponential function to obtain corresponding weights; fusing the fusion features according to the fusion features of each type of modal data and other modal data and the corresponding weights thereof to obtain the fusion features of each type of modal data, and processing the fusion features of each type of modal data by using a multi-head self-attention mechanism; and obtaining an irony detection result of the dialogue according to the fusion features processed by the multi-head self-attention mechanism. According to the invention, effective irony detection is realized, and irony detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for detecting sarcasm in a conversation scenario. Background Art

[0002] Among the existing sarcasm detection methods in conversational scenarios, most studies use a joint recognition method of emotions and feelings to assist in sarcasm detection. However, these methods often fail to fully consider the multimodal and temporal characteristics of human language, resulting in obvious deficiencies in processing actual conversation data.

[0003] First, most existing models fail to effectively handle the problem of data misalignment between different modalities, that is, the misalignment of information caused by differences in feature extraction methods between modalities, which seriously affects the accurate recognition of sarcastic intent. Second, for long-range dependencies in multimodal data, existing technologies often ignore the time series correlation between elements of each modality, making the model ineffective in capturing sarcastic expressions across time points. Finally, emotional inconsistency in cross-modal data is also a major challenge. Current models find it difficult to accurately parse the emotional differences expressed between different modalities, which affects the final sarcasm detection results. Therefore, the application effect of existing models in real multimodal dialogue scenarios needs to be improved. These inherent technical problems need to be effectively solved through more advanced algorithms and model design.

[0004] In view of the above analysis, the technical problems existing in the prior art are:

[0005] (1) Data misalignment: Since each modality uses different channels for feature extraction, there is inherent misalignment between the data.

[0006] (2) Long-range dependency: There is long-range dependency between multimodal elements, which makes traditional word alignment not fully suitable for sarcasm detection.

[0007] (3) Sentiment inconsistency: Sentiment inconsistency in cross-modal data increases the difficulty of identifying sarcastic language. Summary of the Invention

[0008] In order to solve the above technical problems existing in the prior art, the present invention provides a method and system for detecting sarcasm in a dialogue scenario, which can achieve effective sarcasm detection and can effectively detect real-world scenarios.

[0009] The present invention provides a method for detecting sarcasm in a conversation scenario, comprising:

[0010] Extracting features from multimodal data in a conversation to obtain features of each modal data, wherein the multimodal data includes multiple types of text, video, and voice;

[0011] The multi-head cross attention mechanism is used to fuse the features of each modal data with the features of other modal data to obtain fused features. The activation function and normalized exponential function are used in turn to process the fused features of each modal data and other modal data to obtain the corresponding weights.

[0012] According to the fusion features of each modal data and each other modal data and their corresponding weights, the fusion features are fused to obtain the fusion features of each modal data, and the fusion features of each modal data are processed using a multi-head self-attention mechanism;

[0013] A sarcasm detection result of the conversation is obtained based on the fusion features processed by the multi-head self-attention mechanism.

[0014] The present invention also provides a sarcasm detection system in a conversation scenario, comprising:

[0015] An extraction module is used to extract features from multimodal data in the conversation to obtain features of each modal data, where the multimodal data includes multiple types of text, video, and voice;

[0016] The first fusion module is used to use a multi-head cross attention mechanism to fuse the features of each modal data with the features of other modal data to obtain fused features, and sequentially use an activation function and a normalized exponential function to process the fused features of each modal data and each other modal data to obtain corresponding weights;

[0017] The second fusion module is used to fuse the fusion features of each modal data with the other modal data and their corresponding weights to obtain the fusion features of each modal data, and process the fusion features of each modal data using a multi-head self-attention mechanism;

[0018] A detection module is used to obtain a sarcasm detection result of the conversation based on the fusion features processed by the multi-head self-attention mechanism.

[0019] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the sarcasm detection method in any of the above-described dialogue scenarios is implemented.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for detecting sarcasm in any of the above-mentioned dialogue scenarios is implemented.

[0021] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the sarcasm detection method in any of the above-mentioned conversation scenarios.

[0022] The sarcasm detection method and system in conversational scenarios provided by the present invention ensure the quality and richness of input data through data preprocessing and multimodal feature extraction. Cross-modal fusion, through the combination of MHSA and MHCA, enables the model to simultaneously focus on information from different modalities as well as information from its own modality, thereby enhancing the model's overall cross-modal information fusion capabilities and improving its ability to capture complex sarcastic expressions. The final classification step converts feature information into specific sarcasm detection results. When processing sarcastic expressions in everyday conversations, it can more comprehensively understand the diverse forms of sarcasm, thereby improving the accuracy of sarcasm detection. The system has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 This is one of the flow charts of the sarcasm detection method in a conversation scenario provided by the present invention;

[0025] Figure 2 This is the second flow chart of the sarcasm detection method in a conversation scenario provided by the present invention;

[0026] Figure 3 This is an example diagram of sentence alignment in the sarcasm detection method in a conversation scenario provided by the present invention;

[0027] Figure 4 1 is a flow chart of multimodal feature fusion in the sarcasm detection method in a conversation scenario provided by the present invention;

[0028] Figure 5 1 is a schematic diagram of the processing flow of inter-modal attention in the sarcasm detection method in a conversation scenario provided by the present invention;

[0029] Figure 6 1 is a schematic diagram of a data processing flow in a sarcasm detection method in a conversation scenario provided by the present invention;

[0030] Figure 7 4 is a schematic diagram of the structure of the sarcasm detection system in the dialogue scenario provided by the present invention. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0032] The following combination Figure 1 The present invention describes a method for detecting sarcasm in a conversational scenario, comprising:

[0033] Step 101: extracting features from multimodal data in a conversation to obtain features of each modal data, wherein the multimodal data includes multiple types of text, video, and voice;

[0034] Preprocessing and multimodal feature extraction are performed on the multimodal data in the conversation. In this stage, the conversation data is cleaned, labeled, and normalized to ensure the consistency and high quality of the input data.

[0035] Different feature extraction methods are used to extract features from data of different modalities. This embodiment does not limit the modality type and feature extraction method of multimodal data.

[0036] For example, natural language processing techniques (such as word segmentation, part-of-speech tagging, and syntactic analysis) can be used to extract text features from text data. Simultaneously, audio data can be used to extract features such as pitch, speaking rate, and stress. These audio features play an important role in sarcasm detection. Furthermore, video data can be combined to extract features from visual information such as facial expressions and posture to capture potentially sarcastic expressions or actions.

[0037] Step 102: Use a multi-head cross attention mechanism to fuse the features of each modal data with the features of other modal data to obtain a fused feature, and use an activation function and a normalized exponential function in sequence to process the fused features of each modal data and the other modal data to obtain corresponding weights;

[0038] After feature extraction is complete, the cross-modal fusion phase begins. Since sarcasm is often conveyed through multiple modalities, including language, speech, and vision, information from a single modality cannot fully capture the complexity of sarcasm. Therefore, multimodal fusion technology is employed to integrate text, audio, and video features.

[0039] Common fusion methods include attention-based models, recurrent neural networks (RNNs), and long short-term memory (LSTMs). These technologies can establish connections between different modalities, capturing subtle differences in sarcastic expressions and forming a more accurate feature vector for sarcasm identification.

[0040] Step 103: Based on the fusion features of each modal data and each other modal data and their corresponding weights, the fusion features are fused to obtain the fusion features of each modal data, and the fusion features of each modal data are processed using a multi-head self-attention mechanism;

[0041] This embodiment introduces the extracted multimodal features for information exchange. These features first pass through a cross-modal fusion module, which includes a modality-specific feature separator to connect the fusion pipelines of each two modalities. The modality-specific feature separator uses the characteristics of the attention mechanism to perform a preliminary fusion of different modalities to deal with the situation of emotional inconsistency between modalities. Through the inter-modal attention mechanism, the model can pay attention to the complementary information and mutual relationship between different modalities. For example, in the fusion of images and texts, the model identifies the correlation between certain areas in the image and certain words in the text to better understand their interaction. After passing through the inter-modal attention mechanism, each modality has integrated relevant information from other modalities. Then, through the intra-modal attention mechanism, the model strengthens the important features within each modality, helps the model to perform more fine-grained feature extraction within each modality, and improves the representation ability of each modality.

[0042] In the introduction of the cross-modal attention mechanism, α and β represent two different modalities, and and Represent the feature input of different modes respectively. and , l and d represent the sequence length and feature dimension, respectively. This example designs two multi-head attention mechanisms: MHCA (Multi-Head Crossmodal Attention) and MHSA (Multi-Head Self Attention), to capture intra-channel and cross-modal dependencies, respectively.

[0043] The multi-head attention mechanism is implemented using multiple parallel heads, each of which uses proportionally adjusted dot-product attention. Compared with traditional cross-modal fusion methods, it can simultaneously capture sentiment inconsistencies from different modalities and between modalities. This promotes the effective fusion of multi-channel information, significantly improving the accuracy of sarcasm detection.

[0044] Step 104: Obtain a sarcasm detection result of the conversation based on the fusion features processed by the multi-head self-attention mechanism.

[0045] During the sarcasm detection classification phase, sarcasm is detected using a classification algorithm based on the cross-modal fusion feature vector. Common classification models include Support Vector Machines (SVM), Random Forests, and Deep Neural Networks (DNNs). Through training datasets, the model learns the relationship between different modal features and sarcastic expressions. In practice, when new conversation data is input, the model outputs a classification result based on the learned feature patterns, indicating whether sarcasm is present. To improve classification accuracy, the model can also incorporate techniques such as cross-validation and ensemble learning to further optimize sarcasm detection.

[0046] This embodiment achieves sarcasm identification and classification through the three main steps described above. The entire method works by closely integrating these three steps. Data preprocessing and multimodal feature extraction ensure the quality and richness of the input data. Cross-modal fusion, through the combination of MHSA and MHCA, enables the model to simultaneously focus on information from different modalities as well as its own modality, thereby enhancing the model's overall cross-modal information fusion capabilities and improving its ability to capture complex sarcastic expressions. The final classification step converts feature information into specific sarcasm detection results. This method can more comprehensively understand the diverse forms of sarcasm in everyday conversations, thereby improving the accuracy of sarcasm detection and has broad application prospects.

[0047] Based on the above embodiments, Figure 2 As shown, in this embodiment, feature extraction is performed on the multimodal data in the conversation to obtain features of each modal data, including:

[0048] Use the BERTBase model to extract features from the text to obtain features of the text;

[0049] Perform feature extraction on the video using a ResNet-152 model to obtain features of the video;

[0050] The speech is sliced to obtain an audio segment, and OpenSmile is used to perform feature extraction on the audio segment to obtain features of the audio segment.

[0051] This embodiment proposes for the first time a sentence alignment method to perform modal feature alignment. Specifically, the video file is segmented using the end time of each conversation given in the csv file of MUSTARD++. It is worth noting that when a person just starts a conversation, due to the need for shooting techniques, the video may not show the person but the facial expressions of other people or the background. In order to avoid this situation, the end time node of each sentence in the MUSTARD video is used and the image and audio of the frame are cut off. The example diagram of sentence alignment is as follows: Figure 3 shown.

[0052] The MUStARD dataset is a dataset designed for multimodal sarcasm detection in conversations and is very suitable for text feature extraction. The dataset is in English and can provide rich language expressions, especially in terms of subtle language features such as sarcasm and intent. Considering that BERT is a bidirectional model, it can simultaneously utilize contextual information before and after words, and performs particularly well in understanding the semantics and intent in conversations. Therefore, when processing English conversations in the MUStARD dataset, BERT can effectively analyze and extract key features. And BERT has been pre-trained on a large-scale text corpus and can capture a wide range of language knowledge. When processing specific conversation tasks, BERT is fine-tuned to better adapt to the data and requirements of specific tasks. Therefore, this embodiment uses the BERTBase pre-trained model to extract text features as the basis for each utterance. The sentence represents .

[0053] For visual feature extraction, the ResNet-152 pre-trained model can be used to extract the visual features of the end frame of each sentence in the conversation. Since each image of the conversation end frame has been extracted in advance, these images are pre-processed by resizing, center cropping, and normalization. In order to obtain the visual representation of the end time of each sentence, the obtained 2048-dimensional feature vector is calculated and provided. Although there are now more advanced visual feature acquisition strategies, in order to be consistent with the feature extraction of other modal data, the traditional strategy of the same period as other methods is adopted.

[0054] For speech feature extraction, since there may be narration and laughter in the middle of the conversation in the dialogue video, the audio processing library Librosa can be used to remove background noise before feature extraction, and then OpenSmile is used to extract features from the sliced audio segments, and finally provide Audio feature representation.

[0055] Based on the above embodiment, before using the multi-head cross attention mechanism to fuse the features of each modality data with the features of other modality data, this embodiment also includes:

[0056] Use temporal convolution to perform feature convolution operations on each modality data and increase the receptive field.

[0057] In order to ensure that each element of the input sequence has sufficient knowledge of its neighboring elements, and considering the long-range dependencies between multimodal elements, temporal convolution is used to capture long-range dependencies by stacking multiple convolutional layers and increasing the receptive field.

[0058] The receptive field refers to the range of input units corresponding to an output unit in a convolution operation. By increasing the receptive field, temporal convolution can cover input sequences with longer time spans, thereby capturing long-range dependencies.

[0059] The input sequence can be processed by a one-dimensional convolution operation (Conv1D) through a 1D temporal convolution layer to process multimodal data. The formula is as follows:

[0060]

[0061] The three modalities will be represented by L, V and A, representing text, image and audio modalities respectively. Given an input feature sequence and convolution kernel , through one-dimensional convolution operation, the output feature sequence is obtained ,in represents the length of the time series, and d represents the feature dimension.

[0062] Based on the above embodiments, Figure 4 and Figure 5 As shown, in this embodiment, a multi-head cross attention mechanism is used to fuse the features of each modal data with the features of other modal data to obtain fused features, including:

[0063] The features of each other modal data α As a query, the features of the current modality data β As keys and values, the multi-head cross attention mechanism is used to obtain the α feature set focused by the β feature ;

[0064] according to Generate vectors using a positional feedforward network (FFN) , to convert the vector corresponding to the current modal data β Fusion is performed to obtain the fusion features of the modal data α and the current modal data β .

[0065] Since the integration of different modalities is usually affected by the sentiment gap between the hierarchical representations obtained from the pre-trained model, simply combining the information of the three modalities is not enough to effectively model cross-modal inconsistencies, so it performs poorly in achieving the goal of sarcasm detection. Therefore, the inter-modal attention mechanism is introduced. This mechanism aims to utilize the complementarity between the various modalities and capture cross-modal semantic interactions. Since the interaction of the three modalities is very cumbersome to express and there are many repetitions, α and β are used to represent two different modalities, and this is used as an example to demonstrate the inter-modal attention mechanism. The data processing flow is as follows Figure 6 shown.

[0066] Based on the above embodiment, this embodiment fuses the fusion features of each modal data with each other modal data and their corresponding weights using the following formula to obtain the fusion features of each modal data:

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] in, is the normalization layer, For the multi-head cross attention mechanism, is the position feedforward network, and is the dynamically generated attention weight, ; α includes and , When it is text T, and They are video V and voice A respectively; For video V, and They are text T and voice A respectively; For voice A, and They are text T and video V respectively.

[0073] FFN consists of two linear layers separated by ReLU activation function. Taking the current modal data β as text T as an example, the two generated vectors and Fusion is performed and feature weighting is performed using attention weighting , and then further processed by the intra-modal attention layer.

[0074]

[0075] Based on the above embodiments, in this embodiment and The calculation formula is as follows:

[0076]

[0077]

[0078]

[0079]

[0080] in, is the activation function, and is the weight parameter, and is the bias parameter.

[0081] Based on the above embodiment, this embodiment uses a multi-head self-attention mechanism to process the fusion features corresponding to each modality data:

[0082] The fusion features corresponding to each modality data As query, key and value, they are processed by the multi-head self-attention mechanism to obtain the processed fusion features;

[0083] The multi-head self-attention mechanism will pass the query ,key Sum Projecting into H different subspaces and performing scaled dot-product attention enables parallel processing.

[0084] The query ,key Sum They are defined as and .also and represents the learnable parameters of the linear transformation, and is the scaling factor, where n represents the number of heads.

[0085] The multi-head attention mechanism is implemented through n parallel heads, each of which uses proportionally adjusted dot product attention to simultaneously capture the sentiment inconsistency from different modalities and between modalities. The multi-head cross-modal attention mechanism is defined as follows:

[0086]

[0087] In practical applications, the use of MHSA can simultaneously focus on various feature subspaces, thereby achieving more extensive information analysis and utilization. The MHSA mechanism achieves parallel processing by projecting queries, keys, and values into H different subspaces and performing scaled dot product attention. The final output is obtained by concatenating these H results.

[0088] It is important to acknowledge that sarcasm can be expressed within a single modality (whether text or image). To fully capture the subtle and complex meanings of sarcasm in each modality, an intra-modal attention mechanism is employed, which aims to model intra-modal inconsistencies in text or image data. As queries, keys, and values, self-attention mechanisms are used. This technique is used to enhance training stability, speed up convergence, and significantly improve the overall performance of deep neural networks by alleviating internal covariate shift and improving gradient-related complexity, thereby bringing significant improvements during training.

[0089] Finally, we can get a cross-modal representation from α to β: CF stands for Crossmodal Fusion. The combination of MHSA and MHCA enables the model to focus on information from different modalities as well as information from its own modality, thereby enhancing the model's overall cross-modal information fusion capability.

[0090] Based on the above embodiment, this embodiment uses the following formula to process the fusion features corresponding to each modality data using a multi-head self-attention mechanism:

[0091]

[0092]

[0093] in, Modal data obtained after processing by the multi-head self-attention mechanism The corresponding fusion features, is the normalization layer, is the multi-head self-attention mechanism, , MCA is the cross attention mechanism, For the hth , , The hth Corresponding query ,key Sum , Represents the parameter matrix of different heads, is the connection function.

[0094] Based on the above embodiments, in this embodiment, the sarcasm detection result of the conversation is obtained according to the fusion features processed by the multi-head self-attention mechanism using the following formula:

[0095]

[0096] in, is the sarcasm detection result of the dialogue, Indicates a connection operation. and are the trainable parameters of the fully connected layer, 、 and It is the result of processing the fusion features corresponding to the text, video and voice through the multi-head self-attention mechanism.

[0097] The obtained text representation is recorded as , the audio representation is recorded as , and visual representation, denoted as , these representations are concatenated and passed through a fully connected layer with a softmax function to generate a probability distribution in the irony decision space .

[0098] Table 1 shows the MUStARD dataset. By leveraging the visual, textual, and audio information provided by these videos to better detect sarcasm in conversations, it is currently the most widely used and important resource for sarcasm detection. The dataset's annotation process utilizes a custom web interface, enabling accurate sarcasm assessment through contextual clipping. Since the original dataset only contained 345 sarcastic conversations, researchers added an equal number of non-sarcastic conversations to balance the data labels, ultimately creating the MUStARD dataset, consisting of 690 videos. Due to the small amount of data, the data label balance was also maintained when dividing the training, validation, and test sets.

[0099] Table 1 Dataset information table

[0100]

[0101] To verify the feasibility and effectiveness of the proposed model (MulTSD), we compare it with the following state-of-the-art methods:

[0102] BERT: Utilizes the bert-base pre-trained model to extract text features from the target utterance. This is then fed into the softmax function for sarcasm prediction.

[0103] BiLSTM-GloVe: Utilizes the GloVe pre-trained model to extract text features of the target utterance. Softmax is then used to predict sarcasm using the high-level text features obtained after BiLSTM.

[0104] Attn-LSTM: Utilizes the bert-base pre-trained model to extract text features of the target utterance, which is then fed into a text sarcasm detection model based on the attention mechanism and long short-term memory network.

[0105] HFM: A ResNet-50 model pre-trained using fine-tuning extracts image features, and a BiLSTM is used to extract text features. These two extracted features are concatenated as input to the classification layer and finally injected into the Softmax function for sarcasm prediction.

[0106] SVM+Resnet: A method that uses BERT to generate text discourse vectors and feeds them into SVM for sarcasm, sentiment, and emotion analysis.

[0107] RCNN+RoBERTa: This method uses pre-trained RoBERTa vectors to represent sentences and uses RCNN to extract their contextual representations. The final classification is done by a softmax layer.

[0108] EF-Concate: A new sarcasm dataset, the Multimodal Sarcasm Detection Dataset (MUStARD), is used to design a conversational sarcasm detection method.

[0109] A-MTL: Utilizes inter-modal attention (Ie-Attention) and intra-modal attention (Ia-Attention) to link sentences with context, ultimately enabling sarcasm, sentiment, and emotion analysis on the MUStARD dataset.

[0110] ConAttSD: A Contrastive Attention-Based Irony Detection Model (ConAttSD) that exploits the contrastive attention mechanism to extract inter-modal inconsistencies for multimodal irony detection.

[0111] MCER: This method uses the concept of sarcasm and emotion to reproduce 343 incorrect emotion annotations on MUStARD, doubles the size of the dataset, and finally combines emotion labels for sarcasm detection.

[0112] ACMF: Utilizes a multi-channel sentiment inconsistency detection module and a contextual scene inconsistency detection module, and combines them with a cross-channel attention mechanism to achieve sarcasm detection.

[0113] In the experimental setting, the present invention uses the text, image, and audio pre-training models named bert-base, ResNet-152, and ComparE_2016 to extract the feature pre-training models of text, image, and audio respectively.

[0114] The experimental parameters are detailed in Table 2. During training, stochastic gradient descent (ADAMW) was used to solve the objective function, with a learning rate of 0.0001 and a momentum of 0.9. To reduce overfitting during training, the present invention applied the dropout technique with a probability of 0.1 in the attention layer.

[0115] Table 2

[0116]

[0117] In other experimental details, the present invention obtained the checkpoint model with the highest classification accuracy on the test set. The model was trained on RTX4060Ti using the Keras framework and pytorch as the backend.

[0118] Analysis of Unimodal Emotion Recognition Results: To determine the effectiveness of multimodal data for sarcasm detection in conversations, we observe Table 3. We find that sarcasm detection using a single modality is slightly insufficient compared to both bimodal and trimodal methods. However, it is worth noting that some methods exhibit reduced accuracy when comparing bimodal and trimodal methods. We believe this may be due to the presence of noise and background laughter in the audio data set.

[0119] Table 3

[0120]

[0121] Analysis of Multimodal Emotion Recognition Results: The MUStARD dataset allows for experiments in both speaker-dependent and speaker-independent scenarios. However, to better test model performance, some have proposed using a speaker-independent setting to study irony detection. This study only used the context and sarcasm fields in the MUStARD text data. Furthermore, because the goal of this research is to create a system that can detect sarcasm solely from knowledge gained from conversations, rather than from knowledge gained from conversations, only experiments in the speaker-independent setting were conducted.

[0122] Table 4 shows the sarcasm detection results of our optimal model on MUSTARD. Our results significantly outperform all baselines, demonstrating the effectiveness of our dataset processing. The best modal features selected are BART for text, OpenSmile's advanced audio features, and ResNET video features. Furthermore, our method incorporates both inter-modal and intra-modal attention, helping us understand feature importance. Furthermore, while some have used sentiment and emotion labels for sarcasm detection, our method still surpasses their results without them.

[0123] Table 4

[0124]

[0125] Table 5 shows the best results for each modality and modality combination in the speaker-independent setting. Experiments with and without context clearly demonstrate the importance of conversational context in detecting sarcasm in sentences. Although the speaker-dependent setting slightly outperforms all modality combinations, we consider the speaker-independent setting superior because speaker information can bias the system against sarcasm.

[0126] Table 5

[0127]

[0128] The experimental results in Table 3 show that the proposed MulTSD outperforms all baselines on the MUStARD dataset, demonstrating the effectiveness of our model for detecting multimodal sarcasm in conversation. Since single-text models can only classify sentiment on a single-modal dataset, the performance of such methods is relatively low. In contrast, early and mid-stage fusion significantly improve performance, confirming that the combination of visual, textual, and audio content can effectively classify sentiment. As can be seen in Table 3, while existing methods (such as ConAttSD) perform well in detecting multimodal sarcasm in conversation, our MultSD model achieves significant performance improvements across key metrics such as precision, F1 score, and recall. This validates the effectiveness of our approach for detecting multimodal sarcasm in conversation. Furthermore, while A-MTL is not the best among all baselines, its model components are similar to those of ours. However, compared with A-MTL, the performance of the proposed model is significantly improved. This may be due to the effectiveness of the data preprocessing method adopted by the proposed model. During data preprocessing, the proposed model achieves this effect by slicing the video and aligning the three modalities. This also indirectly demonstrates the importance of feature alignment for multimodal sarcasm detection.

[0129] The impact of context on sarcasm detection: Observing Table 5, we find that when there is no context, the model using both text and audio modalities actually outperforms the original trimodal model. Therefore, to explore the importance of context for sarcasm detection results, we conducted the following comparative experiments on TA and TVA. It is worth noting that "Zero" means that the present invention only uses the target utterance and ignores the context. "One" means that the present invention only uses one historical utterance to calculate the intra-modal attention value. "Two" means that the present invention uses the first two historical utterances to calculate the intra-modal attention value, and "All" means that the present invention uses all historical utterances to calculate the intra-modal attention value.

[0130] From Table 6, we can see that no matter still Both perform worst in the zero-context case, demonstrating the importance of modeling conversational context, and both only improve slightly after having one context, indicating that considering only one conversational context is not enough to improve subjective analysis and may introduce noise that interferes with the performance of the multi-task learning framework. However, MultSD's performance in the two-context case is significantly improved compared to the previous result based on itself, regardless of whether still Improvements in precision and recall were 3.13% and 3.15% and 3.5% and 3.66%, respectively. This suggests that considering two contextual sentences is a better option and proves that two historical sentences can provide effective information. Therefore, we hypothesize that combining all conversation contexts can achieve optimal performance, and experimental results confirm this conclusion.

[0131] Table 6

[0132]

[0133] Table 3 clearly shows that our method (OUR) significantly outperforms the baseline model in all three metrics: precision (P), F1 score, and recall (R), demonstrating its effectiveness in sarcasm detection. As shown in the ablation experiment results in Table 5, our method achieves the best performance in sarcasm detection using the fusion of three modalities, both compared to bimodal fusion and single-modality methods. This demonstrates that our modal fusion method can effectively extract multimodal information.

[0134] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0135] First, the present invention designs a sarcasm detection model (MulTSD) based on a multimodal attention mechanism to solve the above problems in an end-to-end manner. In order to achieve data alignment and reduce the influence of long-range dependencies, this paper divides the video into several parts using the video slicing time points given in MUStARD++, and proposes a sentence alignment method. The features are then input into a temporal convolution to reduce the influence of long-range dependencies of multimodal elements. MulTSD adopts both inter-modal and intra-modal attention mechanisms, which can promote interaction between multimodal sequences in the case of modal sentiment inconsistency and adaptively convert the flow of one modality to another. Experimental results on the dataset MUSTARD confirm the superiority of MulTSD. Compared with existing models, it becomes a new state-of-the-art method in sarcasm detection in conversation scenarios. In addition, empirical analysis shows that the cross-modal attention mechanism proposed in MulTSD can capture relevant (1) The present invention designs a sarcasm detection method in conversation scenarios, which divides the video into several parts using the video slicing time points given in MUStARD++ to align multimodal data.

[0136] (2) The present invention inputs features into a temporal convolution and introduces a sentence alignment method to reduce the impact of long-range dependencies of multimodal elements.

[0137] (3) MulTSD adopts both inter-modal and intra-modal attention mechanisms, which can enhance the interaction between multimodal sequences in the presence of modal sentiment inconsistency and adaptively convert the flow of one modality to another.

[0138] (4) The effectiveness of the detection method is evaluated on the MUStARD dataset. The experimental results show that the method not only significantly improves the detection accuracy, but also achieves good results in the test of interpretability of cross-modal signals and more effectively captures the essence of sarcasm.

[0139] Second, as auxiliary evidence for the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0140] (1) The conversation sarcasm detection model of the present invention has significant commercial value in improving the quality and efficiency of customer service. In the application of automated customer service systems, especially chatbots, the present invention can significantly improve the accuracy of the system's understanding of customer intentions and emotions. By accurately identifying the sarcastic content in the conversation, the present invention can effectively reduce communication errors caused by misunderstanding customer intentions and provide customers with solutions that meet their needs in a timely manner, thereby significantly improving customer satisfaction and loyalty. In addition, the application of the present invention can optimize customer service processes, reduce the company's operating costs, and improve the work efficiency of the service team. By reducing the workload of the customer service team, the company can focus more resources and attention on improving the quality of products and services, thereby enhancing the company's market competitiveness and brand image. Therefore, the present invention not only has a positive impact on improving the direct efficiency and effectiveness of customer service, but also brings indirect economic benefits and market advantages to the company. It is a technological innovation with high commercial application value.

[0141] (2) This paper uses an innovative method for detecting sarcasm in conversations to achieve efficient extraction and fusion of image, text, and audio features, significantly improving the accuracy and generalization of detection. It utilizes two attention mechanisms, Multi-Head CrossmodalAttention and Multi-Head Self-Attention, to address the situation of emotional inconsistency between modalities. It also proposes an innovative feature dimension alignment method and temporal convolution for sentence alignment, effectively addressing the challenges of traditional multimodal methods in long-range dependencies and information consistency of multimodal elements. Experimental results show that this paper achieves excellent precision and F1 scores on the MUStARD dataset, overcoming the limitations of single-modality detection methods and demonstrating its great potential and broad prospects in practical applications.

[0142] Third, the sarcasm detection method in a conversational scenario proposed in the present invention solves multiple key problems in the existing technology through steps such as multimodal feature extraction, cross-modal fusion, and classification of sarcasm detection results, and has achieved significant technological progress in industrial applications.

[0143] 1. Technical problems solved:

[0144] Lack of accuracy and comprehensiveness in sarcasm detection: Traditional sarcasm detection methods typically rely on a single modality (such as text) for analysis, making it difficult to accurately capture the complex multimodal expressions of sarcasm, resulting in insufficient detection accuracy and comprehensiveness. This paper introduces multimodal feature extraction and cross-modal fusion technology to effectively address the diversity of sarcasm and improve detection accuracy.

[0145] Integrating multimodal information for sarcasm is difficult: Existing technologies present significant challenges in integrating modal information, including text, audio, and video. This is particularly challenging when processing the correlation between different modalities. Traditional methods fail to fully utilize multimodal information. This paper utilizes technologies such as attention mechanisms, recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) to effectively integrate cross-modal information and enhance sarcasm detection capabilities.

[0146] Inadequate real-time sarcasm detection in conversational scenarios: In practical applications, sarcasm detection needs to be both real-time and efficient. However, existing technologies struggle to quickly and accurately detect sarcasm in real-time conversational scenarios. This paper improves the real-time performance of sarcasm detection by optimizing feature extraction and classification algorithms, enabling rapid response and identification of sarcasm in conversational scenarios.

[0147] 2. Significant technological progress achieved:

[0148] Deep fusion and analysis of multimodal features: This invention realizes the deep fusion of multimodal features such as text, audio and video for the first time in sarcasm detection, breaking through the limitations of single modality detection and enabling the system to more accurately capture the complex information in sarcastic expressions, thereby greatly improving the accuracy of detection.

[0149] Cross-modal fusion technology based on advanced neural network models: This invention adopts a deep neural network (such as RNN, LSTM) model based on the attention mechanism to efficiently fuse different modal features, enhance the ability to understand sarcastic expressions, and significantly improve the detection performance of the system.

[0150] Enhanced Adaptability and Scalability: Thanks to its modular design and advanced machine learning algorithms, the sarcasm detection method is highly adaptable and scalable. This method is applicable not only to conventional conversations but also to a variety of practical scenarios, such as social media comments and customer service conversations.

[0151] Broad prospects in industrial applications: The sarcasm detection method of the present invention has broad prospects in industrial applications, especially in the fields of intelligent customer service, social media analysis, etc. By accurately detecting sarcastic expressions, it can help companies better understand user emotions, optimize customer service, and improve user experience.

[0152] In summary, the present invention has achieved significant technical progress in terms of the accuracy and real-time performance of sarcasm detection, as well as the ability to comprehensively analyze multimodal information. It has solved many problems in the existing technology and provided a new technical means for sentiment analysis and user experience improvement in industrial applications.

[0153] The following describes a sarcasm detection system in a conversation scenario provided by the present invention. The sarcasm detection system in a conversation scenario described below and the sarcasm detection method in a conversation scenario described above can be referenced to each other.

[0154] like Figure 7 As shown, the system includes an extraction module, a first fusion module, a second fusion module and a detection module, wherein:

[0155] An extraction module is used to extract features from multimodal data in the conversation to obtain features of each modal data, where the multimodal data includes multiple types of text, video, and voice;

[0156] The first fusion module is used to use a multi-head cross attention mechanism to fuse the features of each modal data with the features of other modal data to obtain fused features, and sequentially use an activation function and a normalized exponential function to process the fused features of each modal data and each other modal data to obtain corresponding weights;

[0157] The second fusion module is used to fuse the fusion features of each modal data with the other modal data and their corresponding weights to obtain the fusion features of each modal data, and process the fusion features of each modal data using a multi-head self-attention mechanism;

[0158] A detection module is used to obtain a sarcasm detection result of the conversation based on the fusion features processed by the multi-head self-attention mechanism.

[0159] (1) This invention can be integrated into various customer service platforms, such as online customer service systems and intelligent customer service robots. In these systems, the conversational sarcasm detection model can help identify sarcastic language that consumers may use during communication, thereby more accurately understanding and responding to customer needs. The application of this technology can significantly improve customer satisfaction, optimize user experience, and reduce misunderstandings and unnecessary service costs.

[0160] (2) Social platforms can use this invention to monitor and manage user-generated content, especially to automatically identify and filter inappropriate and sarcastic comments. This not only helps to create a healthy online communication environment, but also helps platforms comply with relevant laws and regulations and reduce legal risks caused by content issues.

[0161] (3) In environments that require strict language review, such as public broadcasting or news releases, the present invention can be used as an automated tool to ensure the compliance of published content and prevent misunderstandings or disputes caused by linguistic sarcasm.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for detecting sarcasm in a conversational context, characterized in that: include: Extracting features from multimodal data in a conversation to obtain features of each modal data, wherein the multimodal data includes multiple types of text, video, and voice; The multi-head cross attention mechanism is used to fuse the features of each modal data with the features of other modal data to obtain fused features. The activation function and normalized exponential function are used in turn to process the fused features of each modal data and other modal data to obtain the corresponding weights. According to the fusion features of each modal data and each other modal data and their corresponding weights, the fusion features are fused to obtain the fusion features of each modal data, and the fusion features of each modal data are processed using a multi-head self-attention mechanism; A sarcasm detection result of the conversation is obtained based on the fusion features processed by the multi-head self-attention mechanism.

2. The method for detecting sarcasm in a conversational context according to claim 1, wherein: Perform feature extraction on the multimodal data in the conversation to obtain the features of each modal data, including: Using the BERTBase model to extract features from the text to obtain features of the text; Perform feature extraction on the video using a ResNet-152 model to obtain features of the video; The speech is sliced to obtain an audio segment, and OpenSmile is used to perform feature extraction on the audio segment to obtain features of the audio segment.

3. The method for detecting sarcasm in a conversational context according to claim 1, wherein: Before using the multi-head cross attention mechanism to fuse the features of each modality data with the features of other modal data, it also includes: Use temporal convolution to perform convolution operations on the features of each modality data and increase the receptive field.

4. The method for detecting sarcasm in a conversational context according to claim 1, wherein: The multi-head cross attention mechanism is used to fuse the features of each modality data with the features of other modal data to obtain fused features, including: The features of each other modal data α As a query, the features of the current modality data β As keys and values, the multi-head cross attention mechanism is used to obtain the α feature set focused by the β feature ; according to Generate vectors using a position feed-forward network , to convert the vector corresponding to the current modal data β Fusion is performed to obtain the fusion features of the modal data α and the current modal data β .

5. The method for detecting sarcasm in a conversational context according to claim 4, wherein: The fusion features of each modal data and each other modal data and their corresponding weights are fused by the following formula to obtain the fusion features of each modal data: ; ; ; ; ; in, is the normalization layer, For the multi-head cross attention mechanism, is the position feedforward network, and is the attention weight, ; α includes and , When it is text T, and They are video V and voice A respectively; For video V, and They are text T and voice A respectively; For voice A, and They are text T and video V respectively.

6. The method for detecting sarcasm in a conversational context according to claim 5, wherein: and The calculation formula is as follows: ; ; ; ; in, is the activation function, and is the weight parameter, and is the bias parameter.

7. The method for detecting sarcasm in a conversational context according to claim 1, wherein: Use the multi-head self-attention mechanism to process the fusion features corresponding to each modality data: The fusion features corresponding to each modality data As query, key and value, they are processed by the multi-head self-attention mechanism to obtain the processed fusion features; The multi-head self-attention mechanism is implemented by ,key Sum Projecting into H different subspaces and performing scaled dot-product attention enables parallel processing.

8. The method for detecting sarcasm in a conversational context according to claim 7, wherein: The fusion features corresponding to each modality data are processed using the multi-head self-attention mechanism through the following formula: ; ; in, Modal data obtained after processing by the multi-head self-attention mechanism The corresponding fusion features, is the normalization layer, is the multi-head self-attention mechanism, , MCA is the cross attention mechanism, For the hth , , The hth Corresponding query ,key Sum , Represents the parameter matrix of different heads, is the connection function.

9. The method for detecting sarcasm in a conversation scenario according to any one of claims 1 to 8, wherein: The sarcasm detection result of the conversation is obtained by the following formula based on the fusion features processed by the multi-head self-attention mechanism: ; in, is the sarcasm detection result of the dialogue, Indicates a connection operation. and are the parameters of the fully connected layer, 、 and It is the result of processing the fusion features corresponding to the text, video and voice through the multi-head self-attention mechanism.

10. A sarcasm detection system in a conversational setting, characterized in that: include: An extraction module is used to extract features from multimodal data in the conversation to obtain features of each modal data, where the multimodal data includes multiple types of text, video, and voice; The first fusion module is used to use a multi-head cross attention mechanism to fuse the features of each modal data with the features of other modal data to obtain fused features, and sequentially use an activation function and a normalized exponential function to process the fused features of each modal data and the other modal data to obtain corresponding weights; The second fusion module is used to fuse the fusion features of each modal data with each other modal data and their corresponding weights to obtain the fusion features of each modal data, and use a multi-head self-attention mechanism to process the fusion features of each modal data; A detection module is used to obtain a sarcasm detection result of the conversation based on the fusion features processed by the multi-head self-attention mechanism.

Images