Social media irony semantic recognition method and system, storage medium and electronic device

Abstract

The application provides a social media irony semantic recognition method, system, storage medium and electronic equipment, and relates to the field of irony semantic recognition.In the application, social media multi-modal data is first integrated, and then irony-specific features are activated based on a pre-constructed hierarchical irony semantic library;next, text guidance features, image guidance features, numerical guidance features and irony-specific features are spliced to obtain a global guidance vector, and the global guidance vector is used as an attention query matrix to obtain bidirectional interaction features between the text and the image by using bidirectional attention, to obtain irony perception features by using irony category attention, and to obtain multi-modal fusion features by using additive attention, and a fusion intermediate vector is obtained after splicing;finally, it is identified whether the social media multi-modal data is irony expression.The application realizes the collaborative work of multi-attention mechanism through the cascaded transmission of the global guidance vector combined with the irony-specific features, and significantly improves the recognition ability for complex social media irony scenarios.

Description

Technical Field

[0001] This invention relates to the field of irony semantic recognition, and specifically to a social media irony semantic recognition method, system, storage medium, and electronic device. Background Technology

[0002] Semantic recognition is one of the core tasks in Natural Language Processing (NLP), aiming to understand semantic information in text or speech. The use of rhetorical devices presents significant challenges to NLP; irony is one such commonly used rhetorical device.

[0003] In related technologies, patent application CN119513278A discloses a residual-based multimodal Chinese irony recognition method that integrates language features. The method has the following steps: (1) Using the chi-square statistical method, words with ironic and non-ironic meanings are extracted from the self-constructed image-text multimodal Chinese irony recognition dataset to construct a language feature system; (2) The language features of the words in the language feature system are extracted using the TextCNN model; (3) Image features are extracted using the ResNet34 model; (4) A cross-attention mechanism is introduced to establish the association between image and text features; (5) Feature fusion is performed to obtain the classification result.

[0004] However, the above method relies on fixed residual connection splicing features, which cannot recognize deep-seated ironic semantics. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a method, system, storage medium, and electronic device for recognizing social media irony, solving the technical problem that relying on fixed residual connection splicing features cannot identify deep-level irony.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for recognizing social media irony includes:

[0010] Acquire and preprocess social media multimodal data containing text, images, and numerical data to construct corresponding text-guided features, image-guided features, and numerical-guided features;

[0011] Based on a pre-built hierarchical irony semantic lexicon, the irony-specific features of the pre-processed social media multimodal data are activated; wherein the irony-specific features are one or any combination of the following: expectation asymmetry features, emphasis features, contextual inconsistency features, black humor features, and cultural context features.

[0012] The global guidance vector is obtained by concatenating the text guidance features, image guidance features, numerical guidance features, and irony-specific features; and a fusion intermediate vector is obtained using a pre-trained multimodal hybrid interactive attention network, including:

[0013] The global guidance vector is used as the attention query matrix. Bidirectional attention is used to obtain bidirectional interaction features between text and image. Irony category attention is used to obtain irony perception features. Additive attention is used to obtain multimodal fusion features. The bidirectional interaction features, irony perception features and multimodal fusion features are concatenated to obtain the fusion intermediate vector.

[0014] Based on the fused intermediate vector, the ironic semantics of the social media multimodal data are identified.

[0015] Preferably, the hierarchical ironic semantic lexicon includes a basic layer, a rule layer, and a cultural layer; wherein:

[0016] The pre-construction process of the base layer includes:

[0017] Based on the polarity annotations in the NRC sentiment dictionary, we expand hypothetical vocabulary and realistic descriptive words to construct several text structure patterns with ironic semantics.

[0018] The pre-construction process of the rule layer includes:

[0019] The system pre-determines the adverb enhancement coefficients and basic intensity of sentiment words modified by adverbs of different degrees; pre-determines the scene rationality between several pairs of text scene keywords and image scene types; and pre-determines the contradiction categories between text sentiment polarity and image scene polarity.

[0020] The pre-construction process of the cultural layer includes:

[0021] Based on the ConceptNet commonsense knowledge base, several regionally specific ironic expressions were constructed.

[0022] Preferably, the activation process of the expected asymmetry feature includes:

[0023] Based on the preprocessed text data, hypothetical words and realistic descriptive words are retrieved, and corresponding text structure patterns are matched in the hierarchical ironic semantic lexicon.

[0024] If a match is successful, calculate the semantic distance between the real-world descriptive words and the hypothetical words;

[0025] If the semantic distance is greater than the first threshold, the expected asymmetry feature is activated.

[0026] Preferably, the activation process of the emphasized feature includes:

[0027] Based on the preprocessed text data, degree adverbs and the sentiment words they modify are retrieved, and the corresponding adverb enhancement coefficients and sentiment word base strengths are matched in the hierarchical irony semantic lexicon.

[0028] If a match is successful, the sentiment intensity contradiction value is calculated based on the adverb enhancement coefficient and the basic intensity of the sentiment word;

[0029] If the emotional intensity contradiction value is greater than the second threshold, the emphasis feature is activated.

[0030] Preferably, the activation process of the inconsistency feature in the contextual expression includes:

[0031] Based on the preprocessed text and image data, text scene keywords and image scene types are retrieved, and the degree of conformity between the two and the preset scene rationality is calculated through the hierarchical irony semantic lexicon.

[0032] Calculate the scenario conflict score based on the degree of conformity between the two and the preset scenario.

[0033] If the scenario conflict score is less than the third threshold, the scenario expression inconsistency feature is activated.

[0034] Preferably, the activation process of the dark humor feature includes:

[0035] Based on the preprocessed text and image data, the emotional polarity of the text and the scene polarity of the image are analyzed, and the degree of conformity between the two and the preset contradiction category is calculated through the hierarchical irony semantic lexicon.

[0036] Based on the degree of conformity between the two and the preset contradiction category, the black humor conflict score is calculated;

[0037] If the black humor conflict score is greater than the fourth threshold, the black humor feature is activated.

[0038] Preferably, the activation process of the cultural context features includes:

[0039] Based on the preprocessed text and image data, the emotional polarity of the text and the regional scene of the image are analyzed, and the degree of conformity between the two and the preset regional ironic expressions is calculated in the hierarchical ironic semantic lexicon.

[0040] Based on the degree of conformity between the two and the pre-set regional ironic expressions, the cultural conflict score is calculated;

[0041] If the cultural conflict score is greater than the fifth threshold, the cultural context feature is activated.

[0042] Preferably, the bidirectional interaction features include a first interaction feature and a second interaction feature, and the step of acquiring the bidirectional interaction features between text and image using bidirectional attention includes:

[0043] The global guidance vector is used as the attention query matrix Q. Based on the text guidance features, corresponding key vectors and value vectors are generated to calculate the attention weight of the image to the text using the bidirectional attention. The first interaction feature is obtained by weighted fusion.

[0044] The global guidance vector is used as the attention query matrix Q. Key vectors and value vectors are generated based on the image guidance features. The bidirectional attention is used to calculate the text-image attention weights, and the weighted fusion is used to obtain the second interaction features.

[0045] Preferably, if the preprocessed image data contains ironic visual cues, the attention weight of the image to the text is automatically increased; wherein the methods for obtaining the ironic visual cues include:

[0046] The YOLO network is invoked to detect key visual elements in the preprocessed image data, and residual neural networks are used to extract quantifiable polarity correlation features of the elements; wherein the polarity correlation features include the intensity of human emotions and the integrity of the appearance of objects.

[0047] The polarity correlation feature is compared with a baseline threshold. If it is higher than the baseline threshold by a preset ratio, it is determined to be a positive extreme value feature. If it is lower than the baseline threshold by a preset ratio, it is determined to be a negative extreme value feature. The positive or negative extreme value feature is used as the ironic visual cue.

[0048] Preferably, the step of acquiring irony perception features using irony category attention includes:

[0049] The global guidance vector is used as the attention query matrix Q. Key vectors and value vectors are generated based on the irony-specific features, and irony attention weights are calculated using the irony category attention.

[0050] Based on the aforementioned irony attention weights, the text guidance features, image guidance features, and numerical guidance features are weighted and fused to obtain irony perception features.

[0051] Preferably, the step of acquiring multimodal fusion features using additive attention includes:

[0052] The global guidance vector is used as the attention query matrix Q. Based on the text guidance features, image guidance features and numerical guidance features, corresponding key vectors and value vectors are generated to calculate the text modality weights, image modality weights and numerical modality weights respectively using the additive attention.

[0053] Based on a model-agnostic meta-learning algorithm, the modal dominance in preprocessed social media multimodal data is quantified.

[0054] The weight offset is set based on the modality dominance, and the text modality weight, image modality weight and numerical modality weight are adjusted respectively to obtain the multimodal fusion feature through global weighted fusion.

[0055] A social media irony semantic recognition system includes:

[0056] The acquisition and preprocessing module is used to acquire and preprocess social media multimodal data containing text, images and numerical data to construct corresponding text-guided features, image-guided features and numerical-guided features;

[0057] The activation module is used to activate the irony-specific features of preprocessed social media multimodal data based on a pre-built hierarchical irony semantic lexicon; wherein the irony-specific features are one or any combination of the following: expectation asymmetry features, emphasis features, contextual inconsistency features, black humor features, and cultural context features.

[0058] The fusion module is used to concatenate the text guidance features, image guidance features, numerical guidance features, and irony-specific features to obtain a global guidance vector; and to obtain a fusion intermediate vector using a pre-trained multimodal hybrid interactive attention network, including:

[0059] The global guidance vector is used as the attention query matrix. Bidirectional attention is used to obtain bidirectional interaction features between text and image. Irony category attention is used to obtain irony perception features. Additive attention is used to obtain multimodal fusion features. The bidirectional interaction features, irony perception features and multimodal fusion features are concatenated to obtain the fusion intermediate vector.

[0060] The identification module is used to identify the ironic semantics of the social media multimodal data based on the fused intermediate vector.

[0061] A storage medium storing a computer program for social media satire semantic recognition, wherein the computer program causes a computer to perform the social media satire semantic recognition method as described above.

[0062] An electronic device, comprising:

[0063] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing social media irony semantic recognition as described above.

[0064] (III) Beneficial Effects

[0065] This invention provides a method, system, storage medium, and electronic device for recognizing social media irony. Compared with existing technologies, it has the following advantages:

[0066] This invention first integrates multimodal social media data to construct text-guided features, image-guided features, and numerical-guided features. Second, based on a pre-constructed hierarchical irony semantic lexicon, irony-specific features are activated. Third, the text-guided features, image-guided features, numerical-guided features, and irony-specific features are concatenated to obtain a global guidance vector. This global guidance vector is then used as an attention query matrix. Bidirectional attention is used to obtain bidirectional interaction features between text and images, irony category attention to obtain irony perception features, and additive attention to obtain multimodal fusion features. These are then concatenated to obtain a fusion intermediate vector. Finally, the invention identifies whether the multimodal social media data is an ironic expression. This invention, by combining the cascading transmission of the global guidance vector with irony-specific features, achieves the collaborative work of a multi-attention mechanism, significantly improving the ability to identify complex social media irony scenarios. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A block diagram illustrating a social media irony semantic recognition method provided in an embodiment of the present invention;

[0069] Figure 2 A flowchart of a social media irony semantic recognition method provided in an embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] This invention constructs a multimodal hybrid interactive attention network, a hierarchical ironic semantic lexicon (retrieval enhancement dictionary), and a meta-learning dynamic weight adjustment mechanism to achieve the following objectives:

[0072] 1) Fully integrate multi-source information: integrate text, images, numerical modalities (such as rating data) and explicit-implicit sentiment features to capture cross-modal contradictory cues (such as positive text-negative dark humor features in images), breaking through the limitations of existing technologies that only support dual-modal text-image or single-modal text.

[0073] 2) Overcoming cross-cultural irony recognition: Constructing a three-layer irony semantic lexicon based on the multilingual common sense knowledge base (ConceptNet) (basic layer trigger words, rule layer structural contradictions, and cultural layer regional differences) to solve the recognition bias problem of irony expressions between the East and the West (such as the Eastern euphemism 'You are so good at talking' and the Western exaggeration mark 'Bravo').

[0074] 3) Optimize dynamic interaction and weight allocation: Quantify modal dominance through Model-Agnostic Meta-Learning (MAML) (e.g., dynamically adjust weight offset Δα when text is dominant), adaptively strengthen attention allocation for contradictory modalities, and solve the problem of insufficient cross-modal conflict capture caused by existing technologies relying on fixed weights or simple splicing.

[0075] 4) Enhance the generalization ability of irony categories: Design a feature extraction step specific to irony categories (such as the feature of expectation inequality and the feature of excessive modification of degree adverbs) to cover diverse irony types such as black humor and inconsistent context, and solve the problem of low accuracy of existing technology in recognizing complex irony scenarios.

[0076] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0077] Example 1:

[0078] like Figure 1 As shown, this embodiment of the invention provides a method for recognizing social media irony, including:

[0079] S1. Acquire and preprocess social media multimodal data containing text, images, and numerical data to construct corresponding text-guided features, image-guided features, and numerical-guided features;

[0080] S2. Based on a pre-built hierarchical irony semantic lexicon, activate the irony-specific features of the pre-processed social media multimodal data; wherein the irony-specific features are one or any combination of the following: expectation asymmetry features, emphasis features, contextual inconsistency features, black humor features, and cultural context features;

[0081] S3. Concatenate the text guidance features, image guidance features, numerical guidance features, and irony-specific features to obtain a global guidance vector; and use a pre-trained multimodal hybrid interactive attention network to obtain a fused intermediate vector, including:

[0082] The global guidance vector is used as the attention query matrix. Bidirectional attention is used to obtain bidirectional interaction features between text and image. Irony category attention is used to obtain irony perception features. Additive attention is used to obtain multimodal fusion features. The bidirectional interaction features, irony perception features and multimodal fusion features are concatenated to obtain the fusion intermediate vector.

[0083] S4. Based on the fused intermediate vector, identify the ironic semantics of the social media multimodal data.

[0084] This invention, through the cascading transmission of global guidance vectors that combine irony-specific features, enables the collaborative operation of multiple attention mechanisms, significantly improving the ability to recognize irony in complex social media scenarios.

[0085] like Figure 2 As shown, Figure 2 A flowchart of a social media irony semantic recognition method is disclosed. The following will combine... Figure 2 The steps of the above scheme are described in detail:

[0086] In step S1, social media multimodal data containing text, images, and numerical data are acquired and preprocessed to construct corresponding text-guided features, image-guided features, and numerical-guided features.

[0087] This step acquires and preprocesses social media multimodal data, including text, images, and numerical data. Specifically:

[0088] For text data:

[0089] Stop words are removed using Byte Pair Encoder (BPE). Contextual semantic features are generated using the Robustly Optimized BERT Approach (RoBERTa) encoder, and global semantic vectors are extracted using Siamese network (SBERT) pooling techniques. Furthermore, sentiment categories (e.g., anger and happiness) and polarities (e.g., positive and negative) are labeled using the NRC Word-Emotion Association Lexicon (a sentiment analysis-based word association lexicon). Finally, explicit and implicit sentiments are separated using a bidirectional LSTM + attention mechanism, and the difference between the two is calculated to generate sentiment contradiction features. Finally, the contextual semantic features, global semantic vectors, and sentiment contradiction features are concatenated to form the corresponding text guidance features.

[0090] For image data:

[0091] First, the image is resized to 480×480 pixels and divided into 14×14 (or 12×12) sliding window regions. Next, visual features are extracted and summed using ResNet50 (a variant of the ResNe residual network series, containing 50 convolutional layers). Then, a Contrastive Language–Image Pre-training (CLIP) model based on contrastive learning is used to achieve cross-modal alignment between text and image. Simultaneously, a dynamic grid region is used to prioritize the extraction of high semantic density features such as faces and text. Finally, the visual features and high semantic density features are concatenated to form the corresponding image guidance features.

[0092] For numerical data:

[0093] Numerical data is standardized to [0,1] and mapped to a 128-dimensional feature vector through a fully connected layer, serving as the corresponding numerical guidance feature.

[0094] In step S2, the irony-specific features of the pre-processed social media multimodal data are activated based on the pre-built hierarchical irony semantic lexicon.

[0095] This invention innovatively integrates a hierarchical irony semantic lexicon with multimodal feature extraction, thereby enhancing the retrieval of irony category-specific features. The irony-specific features are one or any combination of the following: expectation asymmetry features, emphasis features, contextual inconsistency features, black humor features, and cultural context features.

[0096] Exemplarily, the above hierarchical ironic semantic lexicon includes a basic layer, a rule layer, and a cultural layer; where:

[0097] The pre-construction process of the basic layer includes:

[0098] Based on the polarity annotation in the NRC sentiment lexicon, expand imaginative words (such as ideal, should have, etc.) and realistic descriptors (such as reality, actual, etc.), and construct several text structure patterns of ironic semantics.

[0099] The pre-construction process of the rule layer includes:

[0100] Preset different degrees of adverbs (such as extremely, completely, slightly, etc.) and the adverb enhancement coefficients of the sentiment words they modify (such as extremely corresponding to 2.5, slightly corresponding to 1.2) and the basic intensity of sentiment words (such as setting the range [0,5]); preset the scene rationality between several pairs of text scene keywords and image scene types; and preset the contradiction categories of text sentiment polarity and image scene polarity, such as the text sentiment polarity being positive and the image scene polarity being negative.

[0101] The pre-construction process of the cultural layer includes:

[0102] Based on the ConceptNet common sense knowledge base, construct several regional ironic expressions, such as the exaggerated irony of the Western English word Bravo (usually translated as 'excellent') paired with a negative image, and the euphemistic irony of the Eastern Chinese phrase "You are really good at talking".

[0103] After obtaining the pre-constructed hierarchical ironic semantic lexicon, this step activates the ironic specific features of the pre-processed social media multi-modal data, which are shown as follows

[0104] (1) The activation process of the expected non-equivalence feature includes:

[0105] Based on the pre-processed text data, retrieve imaginative words and realistic descriptors, and match the corresponding text structure patterns in the hierarchical ironic semantic lexicon.

[0106] If the match is successful, calculate the semantic distance between the realistic descriptor and the imaginative word; the calculation process can be represented by the following mathematical formula:

[0107]

[0108] Where, represents the semantic distance, with a range of [0,1], and the larger the value, the greater the semantic difference between the imaginative word and the realistic descriptor; represents the cosine similarity; 、 respectively represent the realistic descriptor and the imaginative word; This is an efficient sentence embedding model that can transform lexical sentences into low-dimensional semantic vectors.

[0109] If the semantic distance is greater than a first threshold, the expected asymmetry feature is activated. The first threshold here is specifically a distance threshold, which can be 0.8 for example.

[0110] (2) The activation process of the emphasized features includes:

[0111] Based on the preprocessed text data, degree adverbs and the sentiment words they modify are retrieved, and the corresponding adverb enhancement coefficients and sentiment word base strengths are matched in the hierarchical irony semantic lexicon.

[0112] If a match is successful, the sentiment intensity contradiction value is calculated based on the adverb enhancement coefficient and the basic intensity of the sentiment word; the calculation process can be expressed by the following mathematical formula:

[0113]

[0114] in, It indicates the contradictory value of emotional intensity, reflecting the amplified emotional effect brought about by the degree adverb; Indicates the basic strength of emotional words, such as happiness is 3 and ecstasy is 5; Indicates the enhancement coefficient of degree adverbs.

[0115] If the emotional intensity contradiction value is greater than the second threshold, the emphasis feature is activated. The second threshold here is specifically an emotional intensity contradiction value threshold, which can be 1 for example.

[0116] (3) The activation process of the inconsistency feature in the contextual expression includes:

[0117] Based on the preprocessed text and image data, text scene keywords and image scene types are retrieved, and the degree of conformity between the two and the preset scene rationality is calculated through the hierarchical irony semantic lexicon.

[0118] Based on the degree of conformity between the two and the preset scenario, the scenario conflict score is calculated; the calculation process can be expressed by the following mathematical formula:

[0119]

[0120] in, Indicates the score for scene conflict; , These respectively represent the degree of conformity between the text scene keywords, image scene types, and the preset scene; , Corresponding to and The weight.

[0121] If the scene conflict score is less than the third threshold, the scene expression inconsistency feature is activated. The third threshold here is specifically a scene conflict score threshold, which can be 0.3 for example.

[0122] (4) The activation process of the black humor feature includes:

[0123] Based on the preprocessed text and image data, the emotional polarity of the text and the scene polarity of the image are analyzed, and the degree of conformity between the two and the preset contradiction category is calculated through the hierarchical irony semantic lexicon.

[0124] Based on the degree of conformity between the two and the preset contradiction category, the black humor conflict score is calculated; the calculation process can be expressed by the following mathematical formula:

[0125]

[0126] in, Indicates the score for black humor conflict; , These represent the degree of conformity between the text sentiment polarity, the image scene polarity, and the preset contradiction category, respectively.

[0127] If the black humor conflict score is greater than the fourth threshold, the black humor feature is activated. The third threshold here is specifically a black humor conflict score threshold, which can be 0.7 for example.

[0128] (5) The activation process of the cultural context features includes:

[0129] Based on the preprocessed text and image data, the emotional polarity of the text and the regional scene of the image are analyzed, and the degree of conformity between the two and the preset regional ironic expressions is calculated in the hierarchical ironic semantic lexicon.

[0130] Based on the degree of conformity between the two and the pre-defined regional ironic expressions, a cultural conflict score is calculated; the calculation process can be expressed using the following mathematical formula:

[0131]

[0132] in, Indicates the score for cultural conflict;

[0133] , These represent the degree of conformity between the text's emotional polarity, the image's regional setting, and the preset regional ironic expression, respectively.

[0134] If the cultural conflict score is greater than the fifth threshold, the cultural context feature is activated. The fifth threshold here is specifically a cultural conflict score threshold, which can be 0.6 for example.

[0135] In step S3, the text guidance features, image guidance features, numerical guidance features, and irony-specific features are concatenated to obtain a global guidance vector; and a fused intermediate vector is obtained using a pre-trained multimodal hybrid interactive attention network.

[0136] This step concatenates the text guidance features, image guidance features, numerical guidance features, and irony-specific features to obtain a global guidance vector for use in the attention query matrix.

[0137] Next, a pre-trained multimodal hybrid interactive attention network is used to obtain the fused intermediate vector. The relevant steps are as follows:

[0138] First, the global guidance vector is used as an attention query matrix, and bidirectional attention is used to obtain bidirectional interaction features between text and images; specifically:

[0139] The bidirectional interaction features include a first interaction feature and a second interaction feature. The method of acquiring bidirectional interaction features between text and images using bidirectional attention includes:

[0140] S101. Using the global guidance vector as the attention query matrix Q, generate corresponding key vectors and value vectors based on the text guidance features, and use the bidirectional attention to calculate the attention weight of the image to the text, and obtain the first interaction feature by weighted fusion.

[0141] S102. Using the global guidance vector as the attention query matrix Q, generate key vectors and value vectors based on the image guidance features, and use the bidirectional attention to calculate the text-image attention weights, and then weightedly fuse them to obtain the second interaction features.

[0142] Specifically, if the preprocessed image data contains ironic visual cues, the attention weight of the image to the text is automatically increased; wherein the methods for obtaining the ironic visual cues include:

[0143] The YOLO network is invoked to detect key visual elements in the preprocessed image data. The ResNet residual neural network is used to extract quantifiable polarity correlation features of the elements. When segmentation is required, a Mask Region-based Convolutional Neural Network (Mask R-CNN) is used. The polarity correlation features include the intensity of human emotions and the integrity of the appearance of objects.

[0144] The polarity correlation feature is compared with a baseline threshold. If it exceeds the baseline threshold by a preset percentage, it is determined to be a positive extreme value feature; if it falls below the baseline threshold by a preset percentage, it is determined to be a negative extreme value feature. The positive or negative extreme value feature is then used as the ironic visual cue. The baseline threshold is a pre-calculated baseline of the conventional polarity of various features based on existing image samples. These features include the mean, standard deviation, etc.

[0145] Second, utilize irony category attention to obtain irony perception features, specifically:

[0146] S101': Using the global guidance vector as the attention query matrix Q, generate key vectors and value vectors based on the irony-specific features, and use the irony category attention to calculate the irony attention weight.

[0147] S102' Based on the irony attention weight, the text guidance features, image guidance features and numerical guidance features are weighted and fused to obtain irony perception features.

[0148] Third, additive attention is used to obtain multimodal fusion features, specifically:

[0149] S101'', Using the global guidance vector as the attention query matrix Q, generate corresponding key vectors and value vectors based on the text guidance features, image guidance features and numerical guidance features, so as to use the additive attention to calculate the text modality weights, image modality weights and numerical modality weights respectively.

[0150] S102''Based on the Model-Independent Meta-Learning (MAML) algorithm, this algorithm quantifies modal dominance in preprocessed social media multimodal data.

[0151] S103''Based on the modal dominance, set the weight offset and adjust the text modal weight, image modal weight and numerical modal weight respectively to obtain the multimodal fusion feature through global weighted fusion.

[0152] For example, when D(text) > 0.7, the text modality dominance mode is triggered, generating a weight offset Δα = [+0.3, [0.2,0] is used to adjust the text modality weights, image modality weights, and numerical modality weights accordingly. This setting dynamically increases the text modality weights and suppresses image interference, enabling the model to significantly outperform fixed-weight fusion schemes in complex scenarios such as "long text sentences and ambiguous images".

[0153] It should be noted that the above three attention operations are processed in parallel, that is, the embodiments of the present invention can simultaneously acquire bidirectional interaction features, irony perception features and multimodal fusion features.

[0154] Finally, the bidirectional interaction features, irony perception features, and multimodal fusion features are concatenated to obtain a fusion intermediate vector. This fusion intermediate vector contains the results of the three attention interactions mentioned above, preserving the original interaction relationships and reinforcement features.

[0155] In step S4, the ironic semantics of the social media multimodal data are identified based on the fused intermediate vector.

[0156] For example, refer to Figure 2 In this step, the fused intermediate vector is passed through a fully connected layer, a ReLU activation function, and a residual connection in sequence to finally output the probability of the ironic semantic classification, with a value range of 0 to 1, where 0 is no irony and 1 is complete irony.

[0157] Thus, this embodiment of the invention completes the entire process of the social media irony semantic recognition method.

[0158] Example 2:

[0159] This invention provides a social media irony semantic recognition system, comprising:

[0160] The acquisition and preprocessing module is used to acquire and preprocess social media multimodal data containing text, images and numerical data to construct corresponding text-guided features, image-guided features and numerical-guided features;

[0161] The activation module is used to activate the irony-specific features of preprocessed social media multimodal data based on a pre-built hierarchical irony semantic lexicon; wherein the irony-specific features are one or any combination of the following: expectation asymmetry features, emphasis features, contextual inconsistency features, black humor features, and cultural context features.

[0162] The fusion module is used to concatenate the text guidance features, image guidance features, numerical guidance features, and irony-specific features to obtain a global guidance vector; and to obtain a fusion intermediate vector using a pre-trained multimodal hybrid interactive attention network, including:

[0163] The global guidance vector is used as the attention query matrix. Bidirectional attention is used to obtain bidirectional interaction features between text and image. Irony category attention is used to obtain irony perception features. Additive attention is used to obtain multimodal fusion features. The bidirectional interaction features, irony perception features and multimodal fusion features are concatenated to obtain the fusion intermediate vector.

[0164] The identification module is used to identify the ironic semantics of the social media multimodal data based on the fused intermediate vector.

[0165] Example 3:

[0166] This invention provides a storage medium storing a computer program for social media satire semantic recognition, wherein the computer program causes a computer to execute the social media satire semantic recognition method as described in Embodiment 1.

[0167] Example 4:

[0168] This invention provides an electronic device, comprising:

[0169] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing social media irony semantic recognition as described above.

[0170] It is understood that the social media irony semantic recognition system, storage medium and electronic device provided in the embodiments of the present invention correspond to the social media irony semantic recognition method provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the method, and will not be repeated here.

[0171] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0172] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing social media irony, characterized in that, include: Acquire and preprocess social media multimodal data containing text, images, and numerical data to construct corresponding text-guided features, image-guided features, and numerical-guided features; Based on a pre-built hierarchical irony semantic lexicon, the irony-specific features of pre-processed social media multimodal data are activated; The ironic specific features mentioned therein are one or any combination of the following features: expectation asymmetry, emphasis, inconsistency in situational expression, black humor, and cultural context. By concatenating the text guidance features, image guidance features, numerical guidance features, and ironic specific features, a global guidance vector is obtained. And using a pre-trained multimodal hybrid interactive attention network to obtain fused intermediate vectors, including: The global guidance vector is used as the attention query matrix. Bidirectional attention is used to obtain bidirectional interaction features between text and image. Irony category attention is used to obtain irony perception features. Additive attention is used to obtain multimodal fusion features. The bidirectional interaction features, irony perception features and multimodal fusion features are concatenated to obtain the fusion intermediate vector. Based on the fused intermediate vector, the ironic semantics of the social media multimodal data are identified; The hierarchical ironic semantic lexicon comprises a base layer, a rule layer, and a cultural layer; wherein: The pre-construction process of the base layer includes: Based on the polarity annotations in the NRC sentiment dictionary, we expand hypothetical vocabulary and realistic descriptive words to construct several text structure patterns with ironic semantics. The pre-construction process of the rule layer includes: The system pre-determines the adverb enhancement coefficients and basic intensity of sentiment words modified by adverbs of different degrees; pre-determines the scene rationality between several pairs of text scene keywords and image scene types; and pre-determines the contradiction categories between text sentiment polarity and image scene polarity. The pre-construction process of the cultural layer includes: Based on the ConceptNet common sense knowledge base, several regional ironic expressions were constructed. The activation process of the expected asymmetric features includes: Based on the preprocessed text data, hypothetical words and realistic descriptive words are retrieved, and corresponding text structure patterns are matched in the hierarchical ironic semantic lexicon. If a match is successful, calculate the semantic distance between the real-world descriptive words and the hypothetical words; If the semantic distance is greater than the first threshold, the expected asymmetry feature is activated; The method of acquiring multimodal fusion features using additive attention includes: The global guidance vector is used as the attention query matrix Q. Based on the text guidance features, image guidance features and numerical guidance features, corresponding key vectors and value vectors are generated to calculate the text modality weights, image modality weights and numerical modality weights respectively using the additive attention. Based on a model-agnostic meta-learning algorithm, the modal dominance in preprocessed social media multimodal data is quantified. The weight offset is set based on the modality dominance, and the text modality weight, image modality weight and numerical modality weight are adjusted respectively to obtain the multimodal fusion feature through global weighted fusion.

2. The social media irony semantic recognition method as described in claim 1, characterized in that, The activation process of the emphasized features includes: Based on the preprocessed text data, degree adverbs and the sentiment words they modify are retrieved, and the corresponding adverb enhancement coefficients and sentiment word base strengths are matched in the hierarchical irony semantic lexicon. If a match is successful, the sentiment intensity contradiction value is calculated based on the adverb enhancement coefficient and the basic intensity of the sentiment word; If the emotional intensity contradiction value is greater than the second threshold, the emphasis feature is activated; and / or The activation process of the inconsistency feature in the contextual expression includes: Based on the preprocessed text and image data, text scene keywords and image scene types are retrieved, and the degree of conformity between the two and the preset scene rationality is calculated through the hierarchical irony semantic lexicon. Calculate the scenario conflict score based on the degree of conformity between the two and the preset scenario. If the scenario conflict score is less than the third threshold, the scenario expression inconsistency feature is activated. and / or The activation process of the black humor feature includes: Based on the preprocessed text and image data, the emotional polarity of the text and the scene polarity of the image are analyzed, and the degree of conformity between the two and the preset contradiction category is calculated through the hierarchical irony semantic lexicon. Based on the degree of conformity between the two and the preset contradiction category, the black humor conflict score is calculated; If the black humor conflict score is greater than the fourth threshold, the black humor feature is activated. and / or The activation process of the cultural context features includes: Based on the preprocessed text and image data, the emotional polarity of the text and the regional scene of the image are analyzed, and the degree of conformity between the two and the preset regional ironic expressions is calculated in the hierarchical ironic semantic lexicon. Based on the degree of conformity between the two and the pre-set regional ironic expressions, the cultural conflict score is calculated; If the cultural conflict score is greater than the fifth threshold, the cultural context feature is activated.

3. The social media irony semantic recognition method as described in claim 1, characterized in that, The bidirectional interaction features include a first interaction feature and a second interaction feature. The method of acquiring bidirectional interaction features between text and images using bidirectional attention includes: The global guidance vector is used as the attention query matrix Q. Based on the text guidance features, corresponding key vectors and value vectors are generated to calculate the attention weight of the image to the text using the bidirectional attention. The first interaction feature is obtained by weighted fusion. The global guidance vector is used as the attention query matrix Q. Key vectors and value vectors are generated based on the image guidance features. The bidirectional attention is used to calculate the text-image attention weights, and the weighted fusion is used to obtain the second interaction features.

4. The social media irony semantic recognition method as described in claim 3, characterized in that, If the preprocessed image data contains ironic visual cues, the attention weight of the image to the text is automatically increased; The methods for obtaining the aforementioned ironic visual cues include: The YOLO network is invoked to detect key visual elements in the preprocessed image data, and residual neural networks are used to extract quantifiable polarity correlation features of the elements; wherein the polarity correlation features include the intensity of human emotions and the integrity of the appearance of objects. The polarity correlation feature is compared with a baseline threshold. If it is higher than the baseline threshold by a preset ratio, it is determined to be a positive extreme value feature. If it is lower than the baseline threshold by a preset ratio, it is determined to be a negative extreme value feature. The positive or negative extreme value feature is used as the ironic visual cue.

5. The social media irony semantic recognition method as described in claim 1, characterized in that, The method of obtaining irony perception features using irony category attention includes: The global guidance vector is used as the attention query matrix Q. Key vectors and value vectors are generated based on the irony-specific features, and irony attention weights are calculated using the irony category attention. Based on the aforementioned irony attention weights, the text guidance features, image guidance features, and numerical guidance features are weighted and fused to obtain irony perception features.

6. A social media irony semantic recognition system, characterized in that, The method for performing social media irony semantic recognition as described in claim 1 includes: The acquisition and preprocessing module is used to acquire and preprocess social media multimodal data containing text, images and numerical data to construct corresponding text-guided features, image-guided features and numerical-guided features; The activation module is used to activate the irony-specific features of preprocessed social media multimodal data based on a pre-built hierarchical irony semantic lexicon; wherein the irony-specific features are one or any combination of the following: expectation asymmetry features, emphasis features, contextual inconsistency features, black humor features, and cultural context features. The fusion module is used to concatenate the text guidance features, image guidance features, numerical guidance features, and irony-specific features to obtain a global guidance vector; and to obtain a fusion intermediate vector using a pre-trained multimodal hybrid interactive attention network, including: The global guidance vector is used as the attention query matrix. Bidirectional attention is used to obtain bidirectional interaction features between text and image. Irony category attention is used to obtain irony perception features. Additive attention is used to obtain multimodal fusion features. The bidirectional interaction features, irony perception features and multimodal fusion features are concatenated to obtain the fusion intermediate vector. The identification module is used to identify the ironic semantics of the social media multimodal data based on the fused intermediate vector.

7. A storage medium, characterized in that, It stores a computer program for social media satire semantic recognition, wherein the computer program causes a computer to perform the social media satire semantic recognition method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing social media irony semantic recognition as described in any one of claims 1 to 5.

Images