Multimodal sarcasm recognition method, system, storage medium and electronic device

Abstract

The application provides a multi-modal sarcasm recognition method, system, storage medium and electronic equipment, and relates to the field of sarcasm recognition. The application realizes text, image and user star rating three-mode feature extraction and fusion through a large model, combines a light small model to output sarcasm probability and shunts according to a threshold, quickly determines a feature clear sample, triggers another large model driven thought chain reasoning for a mode contradiction and a semantic implicit sample: generates a contradiction clue vector, accumulates reasoning evidence through a parameter non-shared Transform iteration block and checks confidence, and improves implicit sarcasm recognition reliability. The method relies on strong representation and deep reasoning ability of the large model, combines the advantages of efficient operation and quick response of the small model, balances efficiency and accuracy, enhances the robustness of complex scene recognition, adapts to dynamic changes of network expression, reduces the misjudgment risk of typical mode contradiction scene, and updates the small model parameters with high confidence results through knowledge distillation to ensure long-term stability of the recognition ability.

Description

Technical Field

[0001] This invention relates to the field of irony recognition, and specifically to a multimodal irony recognition method, system, storage medium, and electronic device. Background Technology

[0002] With the rapid growth of online content, ironic expressions are widely present in multimodal data such as text, images, and user ratings. Multimodal irony recognition has become an important research direction in the fields of natural language processing and multimodal understanding, and has significant application value in scenarios such as content moderation and public opinion analysis.

[0003] In related technologies, multimodal irony recognition methods often use a single model for end-to-end reasoning, or only fuse text and image dual-modal features, extract features and complete classification through a static network structure, and rely on a one-time forward propagation to achieve the result output.

[0004] However, a single inference architecture cannot achieve a balance between efficiency and accuracy, and it is difficult to effectively identify implicit irony with modal contradictions and semantic ambiguity. At the same time, the parameters of the model are fixed after training, lacking a continuous learning mechanism, making it difficult to adapt to the dynamic evolution of network ironic expressions, resulting in insufficient recognition robustness. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a multimodal irony recognition method, system, storage medium, and electronic device, which solves the technical problems of current multimodal irony recognition technologies, such as difficulty in balancing inference efficiency and recognition accuracy, insufficient ability to handle implicit irony, and inability to adapt to the dynamic changes in network expressions.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a multimodal irony recognition method, comprising: Acquire three-modal data of the content to be identified: text, image, and user star rating. The first major model extracts text features, image features, and rating features respectively, and then fuses the features of each modality to generate multimodal fusion features; The irony probability is output based on the multimodal fusion features using a lightweight small model; When the probability of irony exceeds a preset threshold range, an explicit irony or non-ironic result is directly output; otherwise, the following second model-driven thought chain reasoning step is executed: Using the text features, image features, and rating features, the emotional difference encoding between the text and the image is performed, and the consistency between the star rating and the emotional content of the text and the image is verified, generating a contradiction clue vector; Using the contradictory clue vector as the initial state, the feature transformation is carried out in stages through multiple Transformer iteration blocks with non-shared parameters, and the reasoning evidence is accumulated to output the irony prediction result; Based on the multimodal fusion features and the irony prediction result, a reasonableness check is performed and a confidence score is calculated. If the confidence score is greater than a preset threshold, the current irony prediction result is accepted and output; otherwise, the process returns to the previous step for re-inference.

[0007] Preferably, the multimodal irony recognition method further includes: The irony judgment results with confidence scores greater than a preset threshold are used as enhanced labels. Soft labels are generated through knowledge distillation and a distillation loss is constructed. The parameters of the lightweight small model are then updated through backpropagation.

[0008] Preferably, the step of extracting text features, image features, and rating features through the first large model, and fusing the features of each modality to generate multimodal fusion features includes: The text is tokenized and Transformer encoded to obtain text features; The global semantic features of the image are extracted using a visual model to obtain image features; The star rating is normalized and encoded using a multilayer perceptron to obtain the rating features; The text features and image features are fused using a cross-modal attention mechanism, and the rating features are numerically weighted and fused to generate the multimodal fused features.

[0009] Preferably, each Transformer iteration block is set independently and its parameters are not shared. The hidden state output of the previous iteration block is used as the input of the next iteration block, and multi-step iterative inference is executed serially. Each Transformer iteration block adopts a basic structure of pre-normalization, multi-head self-attention, residual connection, and feedforward network. The preceding iteration block is configured with a first number of attention heads and a feedforward network of a first scale, while the subsequent iteration block is configured with more attention heads than the first number and a feedforward network of a larger scale than the first scale, and introduces cross-modal bias and applies attention sparsity constraints.

[0010] Preferably, the irony prediction result includes an irony score, a non-irony score, and an irony prediction category determined based on both; the step of performing a reasonableness check and calculating a confidence score based on the multimodal fusion features and the irony prediction result includes: The multimodal fusion features and the irony prediction results are concatenated to construct a joint vector, and a reasonableness score is output based on the joint vector through a lightweight verification network. The confidence score is calculated based on the rationality score and the absolute value of the difference between the irony score and the non-irony score.

[0011] Preferably, the irony prediction result includes an irony score, a non-irony score, and an irony prediction category determined based on both; the process of generating soft tags through knowledge distillation includes: A temperature parameter is introduced to soften the irony score and the non-irony score to obtain a soft label; Based on the soft label and the irony probability, a distillation loss is constructed, and the parameters of the lightweight small model are updated through backpropagation.

[0012] Secondly, the present invention provides a multimodal irony recognition system, comprising: The acquisition module is used to acquire three-modal data of the content to be identified: text, image, and user star rating. The extraction and fusion module is used to extract text features, image features, and rating features respectively through the first main model, and to fuse the features of each modality to generate multimodal fused features; The initial judgment module is used to output the irony probability based on the multimodal fusion features using a lightweight small model; The flow control module is used to directly output either explicit irony or non-irony when the irony probability exceeds a preset threshold range; otherwise, it calls the second model-driven thought chain reasoning module, which includes: The backtracking unit is used to encode the emotional differences between the text and the image and to verify the consistency between the star rating and the emotional content of the text and the image by utilizing the text features, the image features and the rating features, and to generate a contradiction clue vector. An iterative unit is used to take the contradictory clue vector as the initial state, transform the features step by step through multiple Transformer iterative blocks with non-shared parameters, accumulate reasoning evidence, and output the irony prediction result. The self-criticism unit is used to perform a reasonableness check and calculate a confidence score based on the multimodal fusion features and the irony prediction result. If the confidence score is greater than a preset threshold, the current irony prediction result is accepted and output; otherwise, the process returns to the previous step for re-reasoning.

[0013] Preferably, the multimodal irony recognition system further includes: The feedback optimization module is used to take the irony judgment results with confidence scores greater than a preset threshold as enhanced labels, generate soft labels through knowledge distillation and construct distillation loss, and backpropagate to update the parameters of the lightweight small model.

[0014] Thirdly, the present invention provides a storage medium characterized in that it stores a computer program, wherein the computer program causes a computer to execute the multimodal irony recognition method as described above.

[0015] Fourthly, the present invention provides an electronic device, the electronic device comprising: Processor and memory; The memory stores program instructions; The processor is configured to run the program instructions to perform the multimodal irony recognition method as described above.

[0016] (III) Beneficial Effects This invention provides a multimodal irony recognition method, system, storage medium, and electronic device. Compared with the prior art, it has the following advantages: This invention utilizes a large model to extract and fuse features from three modalities: text, image, and user star rating. A lightweight small model outputs the probability of irony and distributes it according to a threshold. For samples with clear features, it quickly determines the target. For samples with modal contradictions or semantic ambiguity, it triggers another large model-driven thought chain reasoning: it uses the three modal features to generate contradictory clue vectors, accumulates reasoning evidence through multi-parameter, non-shared Transformer iteration blocks, and verifies confidence levels, thus improving the reliability of implicit irony recognition. This method leverages the strong feature representation and deep reasoning capabilities of the large model, while integrating the efficient computation and fast response of the lightweight small model. It effectively balances reasoning efficiency and recognition accuracy, significantly enhancing the robustness of recognition in complex scenarios such as modal contradictions and semantic ambiguity. It adapts to dynamic changes in network expression, ensuring long-term stability of recognition capabilities and reducing the risk of misjudgment in typical modal contradiction scenarios such as high-allocation negative text / images and low-allocation positive text / images. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a multimodal irony recognition method provided in an embodiment of the present invention; Figure 2 A flowchart of another multimodal irony recognition method provided in an embodiment of the present invention; Figure 3 A collaborative framework diagram of a large model and a small model provided in an embodiment of the present invention; Figure 4 An iterative layer framework diagram provided for an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0019] Component labeling explanation: 100 - electronic device, 101 - memory, 102 - processor, 103 - display. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This invention aims to overcome the shortcomings of existing multimodal irony recognition technologies, such as inefficient and insufficiently deep reasoning architecture, lack of dedicated irony reasoning mechanism, inability of the model to continuously evolve, and imperfect cross-modal interaction mechanism. By constructing a collaborative mechanism between a large model and a lightweight small model, and using thought chain reasoning as the main thread to run through the multimodal implicit irony discrimination process, combined with the lightweight small model to efficiently judge explicit irony, the following objectives are achieved: (1) Overcoming the dual deficiencies of efficiency and depth in reasoning architecture Through a hierarchical reasoning architecture that combines a large language model with a lightweight small model, multimodal samples are first processed by a large model (BERT, ViT, cross-modal attention mechanism) to extract and fuse multimodal features. Then, a lightweight small model is used for rapid judgment. Implicit irony samples that cannot be judged are input into a thought chain-like progressive reasoning mechanism driven by the large language model. The sample is backtracked for verification, iterative reasoning, and self-correction. At the same time, an interpretable discrimination process is generated, which significantly improves the robustness of implicit irony recognition and balances reasoning efficiency and depth.

[0022] (2) Solving the problem of dedicated cognitive reasoning mechanisms for irony We construct a thought chain reasoning mainline adapted to the cognitive logic of irony, and use the thought chain to run through the entire discrimination process of multimodal implicit irony. For multimodal sample data, we sequentially perform backtracking consistency judgment, iterative reasoning and self-criticism. Combined with a lightweight small model, we can quickly judge explicit irony samples, and specifically solve the problem that existing methods do not have a dedicated irony reasoning chain.

[0023] (3) Improve the model's adaptive and continuous evolution capabilities The high-confidence irony discrimination conclusions output by the large language model are used as enhanced labels and continuously fed back to the training set of the traditional small model through knowledge distillation, dynamically updating the parameters of the small model. During the iteration process, the small model gradually learns the complex implicit irony recognition ability of the large model, which can adapt to the continuous evolution of online language and irony patterns, avoid the degradation of model performance over time, and achieve adaptive evolution and long-term performance stability of the model.

[0024] (4) Optimize cross-modal interaction mechanisms Through multimodal feature fusion and consistency verification mechanisms, cross-modal attention mechanisms are used to achieve feature fusion and multimodal information complementarity. In the backtracking consistency stage of the thought chain, a special image-text emotion comparison and scoring verification link is designed to quantitatively analyze the discrepancy between text emotion and visual content and scoring, thereby realizing deep interaction across modalities.

[0025] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0026] Firstly, such as Figure 1 As shown, this embodiment of the invention provides a multimodal irony recognition method, including: S1. Obtain the three-modal data of the content to be identified: text, image, and user star rating. S2. Extract text features, image features, and rating features through the first major model, and fuse the features of each modality to generate multimodal fusion features; S3. Output the irony probability based on the multimodal fusion features using a lightweight small model; S4. When the irony probability exceeds the preset threshold range, directly output either an explicit irony or a non-ironic result; otherwise, execute the following second model-driven thought chain reasoning step: S51. Using the text features, the image features, and the rating features, perform emotion difference encoding between the text and the image, and verify the consistency between the star rating and the emotions of the text and the image, to generate a contradiction clue vector. S52. Using the contradictory clue vector as the initial state, the feature transformation and reasoning evidence are accumulated through multiple Transformer iteration blocks with non-shared parameters, and the irony prediction result is output. S53. Based on the multimodal fusion features and the irony prediction result, perform a reasonableness check and calculate the confidence score. If the confidence score is greater than the preset threshold, accept and output the current irony prediction result; otherwise, return to the previous step for re-inference.

[0027] This method relies on the strong feature representation and deep reasoning capabilities of large models, and integrates the advantages of efficient computation and fast response of lightweight small models. It effectively balances reasoning efficiency and recognition accuracy, and significantly enhances the robustness of recognition in complex scenarios such as modal contradictions and semantic ambiguity. It adapts to the dynamic changes in network expression, ensures the long-term stability of recognition capabilities, and reduces the risk of misjudgment in typical modal contradiction scenarios such as high allocation of negative images and text and low allocation of positive images and text.

[0028] To further adapt to the dynamic changes in network representation and ensure the long-term stability of the model's recognition capabilities, in an optional implementation, such as Figure 2 As shown, the multimodal irony recognition method further includes: S6. Use the irony discrimination results with confidence scores greater than a preset threshold as enhanced labels, generate soft labels through knowledge distillation and construct distillation loss, and backpropagate to update the parameters of the lightweight small model.

[0029] Reference Figure 3 , Figure 3 A collaborative framework diagram for large and small models is disclosed. For example... Figure 2 As shown, this invention proposes a method for identifying irony that combines a large model with a traditional lightweight small model. The method uses thought chain reasoning as the main thread to guide the discrimination process of multimodal implicit irony, while combining it with a small model for rapid inference on explicit irony samples. The large model integrates text, image, and numerical modal data. The thought chain mechanism driven by the large language model sequentially performs backtracking consistency judgment, iterative reasoning, and self-criticism on implicit irony samples, generating interpretable irony discrimination conclusions. The discrimination results of the large language model are then fed back to the small model's training set through knowledge distillation, forming a closed loop of "efficient initial judgment by the small model – accurate inference by the large model – iterative optimization through feedback."

[0030] Specifically, in step S1, the three-modal data of the content to be identified—text, image, and user star rating—are acquired.

[0031] This step acquires three modal data corresponding to the same content to be identified: text, image, and user star rating. The text modal data consists of the text description posted by the user, the image modal data consists of the accompanying visual content, and the star rating modal data consists of the quantitative evaluation of the user's sentiment towards the content. The three types of data are related from the same source and corroborate each other, together forming the multi-dimensional information foundation for irony recognition.

[0032] It should be understood that, compared to related technologies that are mostly limited to bimodal information of text and images, the embodiments of this invention explicitly introduce the user star rating as a key numerical modality as a sentiment criterion, effectively compensating for the limitations of single text and image information in emotional expression. Furthermore, in the subsequent feature fusion stage, a cross-modal attention mechanism and weighted fusion strategy are used to achieve deep complementarity between text, images, and rating information. In the backtracking consistency verification stage of the thought chain, a consistency verification step specifically designed for the rating modality can accurately quantify the degree of emotional deviation in modal contradiction scenarios such as "high-allocation negative text and images" or "low-allocation positive text and images," effectively capturing the deep semantic deviation between literal expression and true intent. This provides a more comprehensive and reliable basis for implicit and veiled irony recognition, significantly improving the accuracy and robustness of irony recognition in complex scenarios.

[0033] In step S2, text features, image features, and rating features are extracted by the first large model, and the features of each modality are fused to generate multimodal fusion features.

[0034] To achieve effective complementarity of multimodal information and improve the comprehensiveness and accuracy of feature representation, in an optional implementation, this step introduces a large model to perform feature extraction operations, including: S21. Tokenize and Transformer encode the text to obtain text features.

[0035] Specifically, for text, the BERT model is used to tokenize sentences, adding special tags such as [CLS] and [SEP], resulting in an input sequence composed of word tokens. The sequence is fed into a Transformer encoder for feature encoding to obtain... .in, For global semantic tagging, The i-th token after text segmentation. Let n be the set of real numbers, n be the actual number of tokens in the text, and d be the feature dimension.

[0036] S22. Use a visual model to extract the global semantic features of the image to obtain image features.

[0037] Specifically, for images, the ViT model is used to extract global semantic features of dimension d. .in, It is a visual encoder.

[0038] S23. Normalize the star rating and encode it using a multilayer perceptron to obtain the rating features.

[0039] Specifically, for the star rating, normalization is performed to map the rating values ​​to the [0,1] interval; then, feature encoding is performed using a multilayer perceptron (MLP) numerical encoder to output rating features of dimension d. .

[0040] Understandably, the feature dimensions of text features, image features, and rating features need to be consistent, and here they are all unified as d.

[0041] S24. The text features and image features are fused using a cross-modal attention mechanism, and the rating features are numerically weighted and fused to generate the multimodal fused features.

[0042] Specifically, text features are first fused through a cross-modal attention mechanism. Image features The image-text fusion features were obtained. Then, a weighting coefficient α (0 < α < 1) is introduced to integrate the image and text features. With rating features Perform weighted fusion to generate multimodal fusion features. .

[0043] in, This is a cross-attention function; the weighting coefficient α is used to balance the weights of image and text features and rating features, and can be flexibly adjusted according to the actual scenario, without being limited to a single value here.

[0044] In step S3, the irony probability is output based on the multimodal fusion features using a lightweight small model.

[0045] In an optional implementation, the lightweight small model employs a bidirectional long short-term memory network. Multimodal feature fusion The input is the BiLSTM network, which outputs hidden layer features y through bidirectional temporal modeling and feature transformation within the network.

[0046] Furthermore, after normalization using the Softmax activation function, the normalized probability distribution is obtained. Finally, the predicted probability (i.e., the irony probability) of the corresponding irony category is output.

[0047] In step S4, when the irony probability exceeds a preset threshold range, an explicit irony or non-irony result is directly output; otherwise, the following second model-driven thought chain reasoning step is executed.

[0048] This step performs a diversion control operation, utilizing the irony probability output by the lightweight small model in step S3 for rapid discrimination: when the irony probability exceeds a preset threshold range, an explicit irony or non-irony result is directly output; when the irony probability exceeds the preset threshold range (corresponding to an implicit irony sample that cannot be judged), in subsequent steps, the implicit irony sample that cannot be judged is input into a thought chain-like progressive reasoning mechanism driven by another large language model, and the sample undergoes backtracking verification, iterative reasoning, and self-correction, while generating an interpretable reasoning path and discrimination criteria. Thus, implicit irony can be deeply semantically parsed, significantly improving the robustness of recognition in complex scenarios, balancing overall reasoning efficiency and discrimination depth.

[0049] It should be noted that the aforementioned preset threshold range is a probability range pre-defined based on actual task requirements and model performance. It is used to distinguish between explicit samples with high confidence and implicit samples with low confidence, achieving efficient data splitting. When the probability of irony is higher than the upper limit of the range, it indicates significant modal contradiction and sufficient evidence, and is directly judged as explicit irony. When the probability of irony is lower than the lower limit of the range, it indicates that the image and text sentiment are consistent and the ratings match, and is directly judged as non-irony. When the probability of irony falls within the range, it indicates that the semantics are obscure and the modal conflict is ambiguous, and is judged as an implicit irony sample, requiring subsequent deep inference.

[0050] Furthermore, it should be understood that the large and small model collaborative architecture proposed in this embodiment of the invention is not a simple combination of large and small models, but rather an intelligent routing based on the difficulty of sample discrimination. The aforementioned diversion control link plays a key role in helping the architecture fundamentally solve the contradiction between the large end-to-end inference computation overhead of a single LLM and the lack of deep inference capability of a static small model, thus achieving a balance between efficiency and depth.

[0051] As mentioned above, when the probability of irony exceeds a preset threshold range, another language model-driven thought chain progressive reasoning mechanism needs to be invoked. This mechanism includes steps S51 to S52, corresponding to a three-stage progressive reasoning process: In step S51, the text features, image features, and rating features are used to encode the emotional differences between the text and the image, and to verify the consistency between the star rating and the emotional content of the text and the image, thereby generating a contradiction clue vector.

[0052] This step corresponds to the backtracking layer of the thought chain-like progressive reasoning mechanism, used to quantify emotional inconsistencies among multiple modalities and extract contradictory clues. Specific implementation details are as follows: First, cross-modal emotion differential coding is used. .

[0053] Secondly, a consistency check is performed between the star rating and the sentiment of the text and images. First, the implicit features are compared again with the sentiment of the text and images, and then the consistency check between the rating and the text and images is performed. .

[0054] Finally, clues are extracted, and conflicting clue vectors are output. This serves as the initial input for subsequent iterative reasoning.

[0055] in, This indicates the difference in features between text and images, representing the degree of inconsistency in the emotional tone of text and images. This indicates the difference between text and rating features, representing the degree of inconsistency between text and rating sentiment. This indicates the difference between image and rating features, representing the degree of inconsistency between image and rating sentiment. This is a feature of inconsistent sentiment between text and images. This indicates whether the rating contradicts the sentiment expressed in the text and images.

[0056] In step S52, with the contradictory clue vector as the initial state, the feature transformation and inference evidence are accumulated through multiple Transformer iteration blocks with non-shared parameters, and the irony prediction result is output.

[0057] This step corresponds to the iterative layer of the thought chain-like progressive reasoning mechanism, using contradictory clue vectors. As the initial state, multiple independent Transformer iteration blocks are used to perform K-step iterative inference to output the ironic prediction result.

[0058] To achieve progressive and in-depth reasoning and enhance the ability to analyze complex implicit irony, in an optional implementation, each Transformer iteration block is set independently and its parameters are not shared. The hidden state output of the previous iteration block is used as the input of the next iteration block, and multi-step iterative reasoning is executed serially.

[0059] Each Transformer iteration block employs a basic structure of Layer Normalization (LN), Multi-Head Self-Attention, residual connections, and feedforward networks.

[0060] Building upon this foundation, the preceding iteration block is configured with a first number of attention heads and a first-scale feedforward network, while the subsequent iteration block is configured with more attention heads and a larger-scale feedforward network. Cross-modal bias is introduced, and attention sparsity constraints are applied. In other words, the complete inference chain is completed sequentially by adjusting the number of multi-head self-attention heads and the intermediate dimensions of the feedforward network. Shallow blocks (pre-sequence) are lightweight with fewer attention heads, focusing on capturing basic features; deep blocks (sub-sequence) are high-dimensional with multi-head attention, focusing on deep semantic modeling. The final output is the prediction result and the predicted logit (raw discriminant score).

[0061] For example, here we take K=3 as an example, such as Figure 4 As shown, the specific operation is as follows: For iteration block 1 (basic contradiction extraction): contradiction clue vector As input, layer normalization (LN) is first performed; then it enters the two-head self-attention sub-layer, and after split-head projection, each head generates its own query vector. Key vector Value vector Calculate attention score according to standard attention formula .in, These are query, key, and value matrices, respectively. The dimension of the key vector. The scaling factor is used. Next, the outputs of each head are concatenated to obtain the enhanced contradictory features. Finally, after residual connections and LN, the data is fed into a two-layer fully connected network (FNN): the first layer uses ReLU activation for nonlinear transformation, and the second layer uses linear transformation, ultimately outputting the hidden state. , To address coarse-grained contradictions, we enhance obvious text-image conflicts and suppress irrelevant noise.

[0062] For Iteration Block 2 (Deep Conflict Modeling): The basic structure is the same as that of Iteration Block 1, the difference being that the number of attention heads is set to 4 to 6, and a cross-modal bias matrix is ​​introduced into the attention score. : This is used to prioritize locations where there are conflicts between image and text features. Furthermore, the FNN is expanded to four layers, outputting the hidden states. , As a feature of deep conflict, it can capture the implicit contradiction between textual satire and image scenes.

[0063] For iteration block 3 (purification of core contradictions): The number of attention heads is set to 6-8, and an attention sparsity constraint is applied to concentrate weights in the core contradictory regions and suppress irrelevant information. Furthermore, the FNN is expanded to six layers, outputting the hidden states. , It exhibits contradictory characteristics of high purity.

[0064] Then, output the hidden state of the last iteration block. The sample is fed into a simple linear classification layer, and two raw discriminant scores are calculated: the irony score. Scoring for non-irony The higher score is taken as the ironic prediction result. (Irony or non-irony), and the final state Prediction results and the original discriminant score , It is passed down to the level of self-criticism.

[0065] In step S53, the rationality of the multimodal fusion features and the irony prediction result is verified and the confidence score is calculated. If the confidence score is greater than the preset threshold, the current irony prediction result is accepted and output; otherwise, the process returns to the previous step S52 for re-inference.

[0066] This step corresponds to the self-criticism layer of the thought chain-like progressive reasoning mechanism. It is used to verify the rationality of the iterative reasoning results and quantify the confidence level. It receives two types of information: the first type is the original multimodal fusion features. The second category is the ironic prediction results given by the iterative layer. and the corresponding two original discriminant scores and Construct a validation branch and calculate the reasonableness score and confidence score.

[0067] To quantify the reliability of reasoning results, provide clear evidence for re-reasoning decisions, and improve the credibility and stability of judgment results, in an optional implementation, the step of performing a reasonableness check and calculating a confidence score based on the multimodal fusion features and the irony prediction results includes: S10. Construct a joint vector by splicing the multimodal fusion features and the irony prediction result, and output a reasonableness score based on the joint vector through a lightweight verification network.

[0068] S20. Calculate the confidence score based on the rationality score and the absolute value of the difference between the irony score and the non-irony score.

[0069] Specifically, the implementation details of this step are as follows: Multimodal fusion features Compared with the prediction results To assemble, first, put the... Convert the data into a simple number (mapping the ironic category to 1 and the non-ironic category to 0); then fuse this value with the multimodal features. The vector is concatenated to construct a joint vector; then, the joint vector is input into a lightweight validation network (a two-layer fully connected network with ReLU activation in the hidden layers) to output a reasonableness score. Where σ is the Sigmoid activation function, used to map the output to the interval [0,1].

[0070] It should be noted that the reasonableness score Used to characterize the degree of matching between the iterative reasoning conclusion and the original multimodal evidence: when the image and text sentiment are consistent and match high scores, but the prediction is ironic, Low; when there is a clear modal contradiction and the prediction is ironic. Too high. The closer to 0, the lower the reliability of the reasoning; the closer to 1, the higher the rationality of the reasoning. Low-rationality samples will be returned to the backtracking layer for re-reasoning.

[0071] Furthermore, the irony score of the iterative layer output is calculated. Scoring for non-irony absolute value of the difference The absolute value of this difference reflects the certainty of the model's prediction; the larger the value, the more certain the model is about the current conclusion.

[0072] Combined with reasonableness score and difference Calculate the final confidence score Compare the confidence score with a preset threshold. Comparison, execution of gating decisions: like If the reasoning is deemed reliable, the current prediction result is accepted, and high-confidence samples are included. An optional feedback optimization mechanism (i.e., knowledge distillation) is incorporated.

[0073] like If the inference is deemed unreliable, the current prediction result is rejected, the sample is returned to the iteration layer, and the number of iterations K is increased to re-infer the inference.

[0074] It should be understood that the embodiments of the present invention design and implement the above-mentioned dedicated thought chain reasoning mechanism for multimodal implicit irony recognition. This mechanism simulates the progressive thinking process of human cognition of irony and structures and computable it. It includes three core stages: Backtracking consistency assessment: This focuses on quantifying the emotional consistency among the three modalities of text, image, and rating. By calculating the contrast between text and image sentiment and verifying the consistency between the rating and the text and image content, vague "feelings are not right" are transformed into quantifiable vectors of contradictory clues, providing a clear starting point and evidence for deep reasoning.

[0075] Iterative Reasoning and Evidence Accumulation: Starting with a vector of contradictory clues as the initial state, multi-step (K-step) reasoning is performed using Transformer iterative blocks. Each step of reasoning re-evaluates the connections between clues through a self-attention mechanism, gradually uncovering and accumulating evidence that supports or opposes the ironic conclusion, forming a reasoning path, and achieving a deep deconstruction of implicit irony.

[0076] Self-criticism and confidence assessment: A validation branch is introduced to evaluate the reasonableness of the initial conclusions derived from iterative reasoning. The prediction confidence is calculated; if the confidence level falls below a threshold, re-inference is triggered or the sample is marked as a difficult case, ensuring the reliability of the judgment results. The entire process generates an interpretable reasoning chain, improving the model's credibility and transparency.

[0077] Preferably, the multimodal irony recognition method further includes a feedback optimization mechanism: In step S6, the irony discrimination results with confidence scores greater than a preset threshold are used as enhanced labels. Soft labels are generated through knowledge distillation and distillation loss is constructed. The parameters of the lightweight small model are then updated through backpropagation.

[0078] To alleviate the problem of model overfitting and simultaneously enable small models to efficiently learn the inference logic of large models, thereby improving the transfer learning effect, in an optional implementation, the process of generating soft labels through knowledge distillation includes: S100. Introduce a temperature parameter to soften the ironic score and the non-ironic score to obtain a soft label; S200. Based on the soft label and the irony probability, construct the distillation loss and backpropagate to update the parameters of the lightweight small model.

[0079] Specifically, the knowledge distillation feedback mechanism introduced in this step is used to transfer the reasoning ability of the large model (the iterative layer of the thought chain) to the lightweight small model, enabling the small model to adaptively optimize and continuously improve the irony recognition performance. The core process consists of three steps: (i) Soft label generation: Extracting the raw predicted scores from the large model output Introducing a temperature parameter T greater than 1 for fractional softening yields soft labels with a smooth probability distribution. To avoid overly extreme prediction results and fully expose the correlation information between categories, (ii) Distillation loss construction: using the ironic probability distribution of the small model output soft labels generated by large models Based on this, a distillation loss function L is constructed using KL divergence to quantify the difference in distribution between the two.

[0080] (iii) Small model parameter update: The distillation loss is calculated based on high-confidence samples and then updated by backpropagation combined with the learning rate. The parameters θ of the lightweight small model are iteratively updated; as the large model continues to infer and output, the small model continuously learns the ability to recognize complex implicit irony, adapts to the dynamic changes in network expression, and achieves long-term performance stability.

[0081] It should be understood that the embodiments of this invention establish the aforementioned knowledge distillation-based, sustainably evolving feedback optimization closed loop, breaking the traditional model's parameter fixation after training. High-confidence discrimination results (especially for difficult samples) obtained by the large language model through the thought chain mechanism are used as "enhanced labels," and continuously fed back to the training process of the lightweight small model through knowledge distillation technology. This allows the small model to continuously learn from the complex reasoning of the large model, gradually mastering the ability to recognize complex patterns such as implicit irony, thereby adapting to the dynamic evolution of online language and irony patterns, and achieving long-term stability and adaptive evolution of model performance.

[0082] To more clearly and intuitively illustrate the technical solutions and execution process of the embodiments of the present invention, the complete processing flow of the multimodal irony recognition method of the present invention will be described in detail below with reference to specific embodiments 1 and 2, from two typical scenarios: rapid identification of explicit irony and accurate inference of implicit irony.

[0083] Example 1 (Rapid Detection of Explicit Irony): 1. Input data Text: "The service today was superb!"; Image: Showing a negative scene of a waiter with a black face and throwing plates; Rating: 1 star.

[0084] 2. Processing flow 2.1 Large Model Feature Extraction and Fusion: BERT is used to tokenize the text, and after concatenating the [CLS] and [SEP] special tags, it is input into the Transformer encoder to output 768-dimensional positive literal sentiment text features; ViT is used to extract global semantic features of the image and output 768-dimensional negative service scene visual features; the 1-star rating is normalized (the normalized value is 0.2) and mapped to 768-dimensional low score numerical features through the MLP numerical encoder.

[0085] First, text features and image features are fused through a cross-modal attention mechanism; then, a weighting coefficient α=0.7 is set to weight and fuse the text-image fusion features and numerical features to generate the final multimodal fusion features.

[0086] 2.2 Fast classification with a small model: The multimodal fusion features are input into the BiLSTM lightweight classifier, and the output classification result is "irony", with a corresponding irony probability of 0.8.

[0087] 2.32 Mind Chain Reasoning Trigger Judgment: This sample is an explicit irony with significant modal contradiction. The irony probability is higher than the preset threshold range. There is no need to trigger the subsequent mind chain reasoning mechanism. The judgment result is directly output.

[0088] Example 2 (Accurate Inference of Implicit Irony): 1. Input data Text: "This product is really good quality (speechless)"; Images: Standard real-life photos of the product with no obvious defects; Rating: 3 stars.

[0089] 2. Processing flow 2.1 Large Model Feature Extraction and Fusion: BERT is used to tokenize the text and concatenate special tags. The Transformer encoder outputs 768-dimensional features to capture the negative tendencies implied within the brackets, forming text features with implicit irony. ViT is used to extract global semantic features of the image, outputting 768-dimensional visual features with no obvious emotional bias and objective neutrality. The 3-star rating is normalized (the normalized value is 0.6) and mapped to 768-dimensional slightly higher score numerical features through an MLP numerical encoder.

[0090] First, text features and image features are fused through a cross-modal attention mechanism; then, a weighting coefficient α=0.7 is set to weight and fuse the text-image fusion features and numerical features to generate the final multimodal fusion features.

[0091] 2.2 Fast classification with small models: The multimodal fusion features are input into the BiLSTM lightweight classifier, and the output irony probability is 0.4, which is lower than the preset threshold range. The result cannot be directly determined, triggering the thought chain progressive reasoning mechanism.

[0092] 2.3 Progressive Reasoning in the Thinking Chain 2.3.1 Retrospective Layer: Integrates three types of features: text, image, and rating, extracts the difference in sentiment between text and image, and performs rating-text consistency verification; detects modal contrast of "implicit negative text, neutral image, and slightly high rating", and generates implicit contradiction clue vector.

[0093] 2.3.2 Iterative Layer: Through three rounds of serial reasoning using Transformer iterative blocks with non-shared parameters, the deep conflict between the satirical tone of the text and the slightly higher score is gradually uncovered, and the implicit satirical features are iteratively refined.

[0094] 2.3.3 Self-criticism layer: verifies the rationality of the reasoning results, and finally outputs the judgment conclusion as "irony", with a confidence level of approximately 0.88.

[0095] 2.4 Feedback Iterative Optimization: A confidence threshold of 0.5 is set. If the confidence of a sample is higher than the threshold, it is considered a high-confidence sample. Soft labels are generated using knowledge distillation technology and used as supervisory signals input into a lightweight small model. Backpropagation updates the model parameters, continuously optimizing the implicit irony recognition capability of the small model.

[0096] It is not difficult to see that the above-mentioned Example 1 is used to illustrate the rapid sorting and efficient discrimination process of explicit irony samples with high confidence and obvious modal contradictions; Example 2 is used to illustrate the deep reasoning, verification and error correction and model iterative optimization process of implicit irony samples with obscure modal contradictions and implicit semantic expressions, thereby fully demonstrating the technical advantages of the multimodal irony recognition method provided by the embodiments of the present invention in balancing reasoning efficiency and recognition accuracy and supporting continuous model evolution.

[0097] Secondly, embodiments of the present invention provide a multimodal irony recognition system, characterized in that it includes: The acquisition module is used to acquire three-modal data of the content to be identified: text, image, and user star rating. The extraction and fusion module is used to extract text features, image features, and rating features respectively through the first main model, and to fuse the features of each modality to generate multimodal fused features; The initial judgment module is used to output the irony probability based on the multimodal fusion features using a lightweight small model; The flow control module is used to directly output either explicit irony or non-irony when the irony probability exceeds a preset threshold range; otherwise, it calls the second model-driven thought chain reasoning module, which includes: The backtracking unit is used to encode the emotional differences between the text and the image and to verify the consistency between the star rating and the emotional content of the text and the image by utilizing the text features, the image features and the rating features, and to generate a contradiction clue vector. An iterative unit is used to take the contradictory clue vector as the initial state, transform the features step by step through multiple Transformer iterative blocks with non-shared parameters, accumulate reasoning evidence, and output the irony prediction result. The self-criticism unit is used to perform a reasonableness check and calculate a confidence score based on the multimodal fusion features and the irony prediction result. If the confidence score is greater than a preset threshold, the current irony prediction result is accepted and output; otherwise, the process returns to the previous step for re-reasoning.

[0098] This system relies on the strong feature representation and deep reasoning capabilities of large models, and integrates the advantages of efficient computation and fast response of lightweight small models. It effectively balances reasoning efficiency and recognition accuracy, and significantly enhances the robustness of recognition in complex scenarios such as modal contradictions and semantic ambiguity. It adapts to the dynamic changes in network expression, ensures the long-term stability of recognition capabilities, and reduces the risk of misjudgment in typical modal contradiction scenarios such as high-allocation negative images and low-allocation positive images and text.

[0099] To further adapt to the dynamic changes in network expression and ensure the long-term stability of the model's recognition capability, in an optional implementation, the multimodal irony recognition system further includes: The feedback optimization module is used to take the irony judgment results with confidence scores greater than a preset threshold as enhanced labels, generate soft labels through knowledge distillation and construct distillation loss, and backpropagate to update the parameters of the lightweight small model.

[0100] To achieve effective complementarity of multimodal information and improve the comprehensiveness and accuracy of feature representation, in an optional implementation, the step of extracting text features, image features, and rating features through a first large model, and fusing the features of each modality to generate multimodal fused features includes: The text is tokenized and Transformer encoded to obtain text features; The global semantic features of the image are extracted using a visual model to obtain image features; The star rating is normalized and encoded using a multilayer perceptron to obtain the rating features; The text features and image features are fused using a cross-modal attention mechanism, and the rating features are numerically weighted and fused to generate the multimodal fused features.

[0101] In order to achieve progressive deepening of the reasoning process and enhance the parsing ability of complex implicit irony, in an optional implementation, each Transformer iteration block is set independently and the parameters are not shared. The hidden state output of the previous iteration block is used as the input of the next iteration block, and multi-step iterative reasoning is executed serially. Each Transformer iteration block adopts a basic structure of pre-normalization, multi-head self-attention, residual connection, and feedforward network. The preceding iteration block is configured with a first number of attention heads and a feedforward network of a first scale, while the subsequent iteration block is configured with more attention heads than the first number and a feedforward network of a larger scale than the first scale, and introduces cross-modal bias and applies attention sparsity constraints.

[0102] To quantify the reliability of reasoning results, provide a clear basis for re-reasoning decisions, and improve the credibility and stability of the judgment results, in an optional implementation, the irony prediction result includes an irony score, a non-irony score, and an irony prediction category determined based on both; the step of performing a reasonableness check and calculating a confidence score based on the multimodal fusion features and the irony prediction result includes: The multimodal fusion features and the irony prediction results are concatenated to construct a joint vector, and a reasonableness score is output based on the joint vector through a lightweight verification network. The confidence score is calculated based on the rationality score and the absolute value of the difference between the irony score and the non-irony score.

[0103] To alleviate the overfitting problem in model predictions and simultaneously enable small models to efficiently learn the inference logic of large models, thereby improving the transfer learning effect, in an optional implementation, the irony prediction result includes an irony score, a non-irony score, and an irony prediction category determined based on both; the process of generating soft labels through knowledge distillation includes: A temperature parameter is introduced to soften the irony score and the non-irony score to obtain a soft label; Based on the soft label and the irony probability, a distillation loss is constructed, and the parameters of the lightweight small model are updated through backpropagation.

[0104] Thirdly, embodiments of the present invention provide a storage medium storing a computer program, wherein the computer program causes a computer to execute the multimodal irony recognition method provided in any embodiment of the present invention.

[0105] In embodiments of the present invention, any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0106] Fourthly, embodiments of the present invention provide an electronic device. Figure 5 The diagram shown is a structural schematic of the electronic device 100 provided in an embodiment of the present invention. In some embodiments, the electronic device may be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or other terminal device. Furthermore, the multimodal irony recognition method provided in this embodiment can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. This embodiment does not limit the specific application scenarios of the multimodal irony recognition method.

[0107] like Figure 5 As shown, the electronic device 100 provided in this embodiment of the invention includes a memory 101 and a processor 102.

[0108] The memory 101 is used to store computer programs; preferably, the memory 101 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0109] Specifically, memory 101 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 101 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0110] The processor 102 is connected to the memory 101 and is used to execute the computer program stored in the memory 101 so that the electronic device 100 executes the multimodal irony recognition method provided in any embodiment of the present invention.

[0111] In an optional implementation, the processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, evaluation index gate or transistor logic devices, or evaluation index hardware components.

[0112] In an optional embodiment of the present invention, the electronic device 100 may further include a display 103. The display 103 is communicatively connected to the memory 101 and the processor 102, and is used to display the relevant GUI interactive interface of the multimodal irony recognition method.

[0113] It is understood that the multimodal irony recognition system, storage medium and electronic device provided in the embodiments of the present invention correspond to the multimodal irony recognition method provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the method, and will not be repeated here.

[0114] In summary, compared with existing technologies, it has the following beneficial effects: 1. Significantly improves the accuracy and robustness of implicit irony recognition: This embodiment of the invention uses a thought chain mechanism to perform deep reasoning on difficult samples, effectively distinguishing genuine ironic intent from simple literal contradictions. Especially for test samples containing implicit irony and subtle contradictions, the F1 score is significantly improved. This demonstrates the crucial role of a dedicated reasoning mechanism in improving the model's discriminative precision.

[0115] 2. While significantly reducing computational overhead, the inference depth of complex samples is guaranteed: The hierarchical collaborative architecture proposed in this embodiment of the invention delegates most simple samples to small models for processing, effectively diverting the computational pressure of large models.

[0116] 3. Enhanced interpretability and transparency of the model's judgment process: The "backtracking-iteration-self-evaluation" reasoning path generated by the thought chain mechanism proposed in this embodiment provides a clear chain of evidence and logical basis for each judgment (especially for difficult samples). For example, when judging a sample, the model can output an explanation such as "The text appears positive, but the image scene is negative, the score is neutral to low, and there is a significant inconsistency among the three. After three steps of iterative reasoning, it tends to be judged as irony." This greatly increases the user's trust in the model's decisions and facilitates algorithm auditing and optimization.

[0117] 4. It endows the model with the ability to continuously learn and adaptively evolve, effectively alleviating the performance degradation problem: The embodiments of this invention use knowledge distillation feedback loop, so that small models can continuously learn from the new judgments of large models.

[0118] 5. By introducing a rating modality and deep consistency verification, the false positive rate in specific scenarios is reduced: especially in scenarios where the text and image emotions are contradictory (such as "text praise paired with a terrible picture"), existing methods are prone to misinterpreting it as strong sarcasm. This invention, through rating modality verification (for example, if the rating is also low, the sarcasm judgment is strengthened; if the rating is high, caution is needed, as it may not be sarcasm), can more accurately grasp the true intent. Test data shows that the false positive rate is reduced in such easily confused scenarios.

[0119] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal irony recognition method, characterized in that, include: Acquire three-modal data of the content to be identified: text, image, and user star rating. The first major model extracts text features, image features, and rating features respectively, and then fuses the features of each modality to generate multimodal fusion features; The irony probability is output based on the multimodal fusion features using a lightweight small model; When the probability of irony exceeds a preset threshold range, an explicit irony or non-ironic result is directly output; otherwise, the following second model-driven thought chain reasoning step is executed: Using the text features, image features, and rating features, the emotional difference encoding between the text and the image is performed, and the consistency between the star rating and the emotional content of the text and the image is verified, generating a contradiction clue vector; Using the contradictory clue vector as the initial state, the feature transformation is carried out in stages through multiple Transformer iteration blocks with non-shared parameters, and the reasoning evidence is accumulated to output the irony prediction result; Based on the multimodal fusion features and the irony prediction result, a reasonableness check is performed and a confidence score is calculated. If the confidence score is greater than a preset threshold, the current irony prediction result is accepted and output; otherwise, the process returns to the previous step for re-inference.

2. The multimodal irony recognition method as described in claim 1, characterized in that, Also includes: The irony judgment results with confidence scores greater than a preset threshold are used as enhanced labels. Soft labels are generated through knowledge distillation and a distillation loss is constructed. The parameters of the lightweight small model are then updated through backpropagation.

3. The multimodal irony recognition method as described in claim 1, characterized in that, The process involves extracting text features, image features, and rating features using the first major model, and then fusing these modal features to generate multimodal fused features, including: The text is tokenized and Transformer encoded to obtain text features; The global semantic features of the image are extracted using a visual model to obtain image features; The star rating is normalized and encoded using a multilayer perceptron to obtain the rating features; The text features and image features are fused using a cross-modal attention mechanism, and the rating features are numerically weighted and fused to generate the multimodal fused features.

4. The multimodal irony recognition method as described in claim 1, characterized in that, Each Transformer iteration block is set independently and its parameters are not shared. The hidden state output of the previous iteration block is used as the input of the next iteration block, and multi-step iterative inference is executed serially. Each Transformer iteration block adopts a basic structure of pre-normalization, multi-head self-attention, residual connection, and feedforward network. The preceding iteration block is configured with a first number of attention heads and a feedforward network of a first scale, while the subsequent iteration block is configured with more attention heads than the first number and a feedforward network of a larger scale than the first scale, and introduces cross-modal bias and applies attention sparsity constraints.

5. The multimodal irony recognition method as described in claim 1, characterized in that, The irony prediction results include an irony score, a non-irony score, and an irony prediction category determined based on the two. The step of performing a reasonableness check and calculating a confidence score based on the multimodal fusion features and the irony prediction results includes: The multimodal fusion features and the irony prediction results are concatenated to construct a joint vector, and a reasonableness score is output based on the joint vector through a lightweight verification network. The confidence score is calculated based on the rationality score and the absolute value of the difference between the irony score and the non-irony score.

6. The multimodal irony recognition method as described in claim 2, characterized in that, The irony prediction results include an irony score, a non-irony score, and an irony prediction category determined based on the two. The process of generating soft tags through knowledge distillation includes: A temperature parameter is introduced to soften the irony score and the non-irony score to obtain a soft label; Based on the soft label and the irony probability, a distillation loss is constructed, and the parameters of the lightweight small model are updated through backpropagation.

7. A multimodal irony recognition system, characterized in that, include: The acquisition module is used to acquire three-modal data of the content to be identified: text, image, and user star rating. The extraction and fusion module is used to extract text features, image features, and rating features respectively through the first main model, and to fuse the features of each modality to generate multimodal fused features; The initial judgment module is used to output the irony probability based on the multimodal fusion features using a lightweight small model; The flow control module is used to directly output either explicit irony or non-irony when the irony probability exceeds a preset threshold range; otherwise, it calls the second model-driven thought chain reasoning module, which includes: The backtracking unit is used to encode the emotional differences between the text and the image and to verify the consistency between the star rating and the emotional content of the text and the image by utilizing the text features, the image features and the rating features, and to generate a contradiction clue vector. An iterative unit is used to take the contradictory clue vector as the initial state, transform the features step by step through multiple Transformer iterative blocks with non-shared parameters, accumulate reasoning evidence, and output the irony prediction result. The self-criticism unit is used to perform a reasonableness check and calculate a confidence score based on the multimodal fusion features and the irony prediction result. If the confidence score is greater than a preset threshold, the current irony prediction result is accepted and output; otherwise, the process returns to the previous step for re-reasoning.

8. The multimodal irony recognition system as described in claim 7, characterized in that, Also includes: The feedback optimization module is used to take the irony judgment results with confidence scores greater than a preset threshold as enhanced labels, generate soft labels through knowledge distillation and construct distillation loss, and backpropagate to update the parameters of the lightweight small model.

9. A storage medium, characterized in that, It stores a computer program, wherein the computer program causes the computer to perform the multimodal irony recognition method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, The electronic device includes: Processor and memory; The memory stores program instructions; The processor is configured to run the program instructions to perform the multimodal irony recognition method as described in any one of claims 1 to 6.

Images