Method and system for detecting multi-modal sarcasm based on large language model vocabulary projection of image-text semantic alignment

Abstract

This invention relates to the field of multimodal information processing technology, specifically to a method and system for multimodal irony detection based on image-text semantic alignment using a large language model's lexical projection. The method involves acquiring and encoding the text and image data to be detected; constructing a cross-modal fusion module, utilizing the pre-trained embedding layer weights of the large language model, and aligning image features to the text semantic space through a multi-head cross-attention mechanism to obtain aligned image features, which are then fused with the text embedding features; inputting the fused features into the large language model and fine-tuning them using an efficient parameter fine-tuning method to obtain high-dimensional semantic features; and using a classification module based on multimodal attention to perform self-attention enhancement and progressive dimensionality compression on the high-dimensional semantic features, outputting the irony detection result. This invention effectively improves the accuracy and robustness of irony detection in scenarios with weak image-text association, and suppresses the overfitting and illusion problems of the large language model.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing technology, specifically to a method and system for detecting multimodal irony based on lexical projection of large language models and semantic alignment of text and images. Background Technology

[0002] With the rapid development of social networks, social media platforms have become core spaces for public expression, emotional exchange, and information acquisition. Users widely employ multimodal methods combining text and images with synchronized audio and video for information interaction. Against this backdrop, satire, as a prevalent rhetorical device, has become increasingly complex and subtle in its manifestations. For example, in political commentary or discussions of breaking news, satire is often used as a tool to express dissatisfaction or attack others, easily leading to public cognitive biases and exacerbating social conflicts. Simultaneously, satire often serves as cover for cyberbullying, with attackers using "jokes" or "irony" to attempt to evade content moderation. Therefore, improving the ability to detect multimodal satire is of great significance for purifying the online environment.

[0003] Currently, the mainstream techniques for multimodal irony detection can be divided into two categories: The first category comprises irony detection methods based on graph neural networks or attention mechanisms. These methods have made significant progress in feature fusion and inconsistency modeling, but most rely on explicit information between images and text for judgment, resulting in insufficient recognition capabilities when faced with relatively subtle ironic expressions. For example, existing research, based on the inherent image-text inconsistency in irony from psychological theory, extracts features at both the factual and emotional levels for detection, but with low accuracy. Subsequent work introduced graph attention dynamic connection components to capture factual differences between images and text, and combined external emotional knowledge to reconstruct weights to capture emotional inconsistencies; however, they still struggle to accurately identify subtle ironic information. A common drawback of these methods is that they typically do not consider prior knowledge such as the cultural and social context upon which irony depends, and their performance is limited on data with weak image-text associations (i.e., weak cross-modal connectivity).

[0004] The second category is irony detection methods based on large language models. Traditional machine learning or deep learning methods are limited by the scale of training data, making it difficult to capture deep semantic connections and contextual information in images and text, resulting in insufficient generalization ability. Large language models, through self-supervised learning on massive corpora, can transfer rich linguistic and world knowledge as prior information to downstream tasks, enhancing complex semantic understanding and cross-modal alignment capabilities. However, existing methods based on large language models still have insufficient understanding of visual modalities. General multimodal models perform poorly in specialized domains such as irony, failing to understand deep ironic meanings. Even after fine-tuning and cue engineering, they are still severely affected by the "illusion problem," leading to output errors or irrelevant information and low F1 scores. Summary of the Invention

[0005] The purpose of this invention is to propose a multimodal irony detection method and system based on the semantic alignment of text and image using large language model lexical projection, in order to solve the problems of missing background information, poor processing ability of weak cross-modal connections, and overfitting caused by the illusion of large language models in the existing technology, thereby improving the accuracy and robustness of irony detection.

[0006] According to a first aspect of the embodiments of this disclosure, a multimodal irony detection method based on lexical projection of a large language model and semantic alignment of text and image is provided, comprising the following steps: Acquire the multimodal data to be detected, wherein the multimodal data includes text data and image data; The text data and the image data are encoded respectively to obtain text embedding features and image embedding features; A cross-modal fusion module is constructed. By utilizing the pre-trained embedding layer weights of a large language model, the image embedding features are aligned to the semantic space of the text embedding features to obtain aligned image features. Then, the aligned image features are fused with the text embedding features to obtain fused features. The fused features are input into a large language model, and the large language model is fine-tuned using a parameter-efficient fine-tuning method to obtain high-dimensional semantic features; A classification module based on multimodal attention is constructed to perform multimodal self-attention enhancement and progressive dimensionality compression on the high-dimensional semantic features, and output the classification result of irony detection.

[0007] In one embodiment, the image embedding features are aligned to the semantic space of the text embedding features using the pre-trained embedding layer weights of the large language model. Specifically, this includes: passing the embedding layer weight matrix of the large language model through a linear layer. Mapping to low-dimensional word prototypes ,in, For the vocabulary dimension, Let be the dimension after linear mapping, and ; The hidden layer dimension of the model; Using a multi-head cross-attention mechanism, the image embedding features are... As a query, with the stated word prototype As keys and values, the aligned image features are obtained: in, P The number of image blocks. For the first i A query matrix with attention heads For the first i The key matrix of each attention head, For the first i The value matrix of each attention head, For the first i The dimension of attention head.

[0008] In one embodiment, in the multi-head cross-attention mechanism, for each attention head... ,definition: , , in, , , Embed the feature matrix into the image. H The total number of attention heads in a multi-head attention mechanism; For the first i One attention head is used to generate the linear transformation weight matrix of the query matrix. For the first i One attention head is used to generate the linear transformation weight matrix of the key matrix. For the first i One attention head is used to generate a linear transformation weight matrix for the value matrix; is the dimension of the weight matrix.

[0009] Output of each head The aligned image features are obtained after stitching. .

[0010] In one embodiment, the fusion feature is obtained by embedding the text features. The aligned image features By concatenating along the feature dimensions, the fused features are obtained. ,in n The number of words.

[0011] In one embodiment, the parameter-efficient fine-tuning method is a low-rank adaptive method, namely: for the pre-trained weight matrix of a large language model Introduce two low-rank matrices and The forward propagation process was modified to Freeze during training Only update parameters and ,in This refers to the pre-trained original weight matrix in the large language model. d The dimension of the output feature is the number of rows in the weight matrix. K The dimension of the input features is the number of columns in the weight matrix. This is the feature vector input to this layer.r Let be the rank of the low-rank decomposition. h This is the final output of this layer.

[0012] In one embodiment, the classification module based on multimodal attention is implemented as follows: First, multimodal self-attention operations are performed on the high-dimensional semantic features to enhance the interaction between the text and image modalities; Then, following a preset decreasing factor, the feature dimension is gradually reduced by passing through a linear layer and a normalization layer in sequence. Finally, the fully connected layer maps the result to the binary classification output.

[0013] In one embodiment, when encoding the image data, the block embedding module of the Vision Transformer (ViT) is used, and the image is subjected to layer normalization processing.

[0014] According to a second aspect of the present disclosure, a multimodal irony detection system based on large language model lexical projection and text-text semantic alignment is provided, comprising: The data acquisition module acquires the multimodal data to be detected, including text data and image data; The feature encoding module encodes the text data and the image data respectively to obtain text embedding features and image embedding features; The cross-modal fusion module utilizes the pre-trained embedding layer weights of the large language model to align the image embedding features to the semantic space of the text embedding features, obtaining aligned image features; then, the aligned image features are fused with the text embedding features to obtain fused features. The large language model fine-tuning module inputs the fused features into the large language model and uses a parameter-efficient fine-tuning method to fine-tune the large language model to obtain high-dimensional semantic features; The classification output module constructs a classification module based on multimodal attention, performs multimodal self-attention enhancement and progressive dimensionality compression on the high-dimensional semantic features, and outputs the classification result of irony detection.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the aforementioned multimodal irony detection method based on large language model lexical projection for image-text semantic alignment.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the aforementioned multimodal irony detection method based on large language model lexical projection with text-to-image semantic alignment.

[0017] The VTA-Sarc multimodal sarcasm detection model based on a large language model proposed in this invention has the following beneficial effects: (1) This invention integrates aligned image features into the model training and inference process, making full use of the rich pre-training context knowledge of the large language model for multimodal analysis, which significantly improves the model's ability to understand the semantic relationship between images and text.

[0018] (2) The present invention designs a low-rank decomposition classification module to replace the traditional single-layer classification head, which effectively alleviates the information loss and overfitting problems caused by direct dimensionality reduction of high-dimensional features, and enhances the robustness and classification accuracy of the model.

[0019] (3) This invention systematically explores the fine-tuning strategy of large language model during training. Optimizing the overall self-attention module can achieve relatively excellent performance, providing an efficient and reliable solution for multimodal irony detection tasks. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0021] Figure 1 Schematic diagram of VTA-Sarc, a multimodal irony detection method for image-text semantic alignment; Figure 2 This is a diagram showing the results of case analysis in the implementation method; Figure 3 Visualizing attention-based heatmaps; Figure 4 A schematic diagram illustrating the impact of different LoRA hyperparameter settings on MSDLLM performance. Detailed Implementation

[0022] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0026] Example 1: like Figure 1 As shown, this embodiment provides a visual-text semantic alignment via LLM vocabulary projection for multi-modal sarcasm detection (VTA-Sarc). This method fully utilizes a large language model for sarcasm detection, effectively solving the problem of missing background information. Specifically, for the weak modal connectivity problem, this method designs cross-modal fusion, deeply fusing text and visual information through the pre-trained weights of the large language model, enabling the model to more effectively utilize pre-trained background knowledge to judge sarcasm information, thereby overcoming the problem that existing cross-modal alignment strategies are too simplistic. Simultaneously, to address the problem of high output dimensionality and susceptibility to hallucination interference from large language models, a low-rank decomposition classification is designed. By adjusting the feature space mapping, the model complexity is reduced, overfitting is suppressed, thereby improving the model's generalization ability and robustness, and avoiding the influence of erroneous or irrelevant generated information. The specific steps include: S1. Acquire the multimodal data to be detected, wherein the multimodal data includes text data and image data; Specifically, the text data can originate from user posts, comments, or titles on social media platforms, typically in the form of short text in natural language; the image data is the accompanying image corresponding to the text, and can be in common image formats such as JPEG and PNG. It should be noted that there should be a semantic connection between the text and the image, meaning that they together constitute a complete multimodal sample.

[0027] S2. Encode the text data and the image data respectively to obtain text embedding features and image embedding features; To enable the large language model to understand text and lay the foundation for subsequent image feature alignment, this invention uses a pre-trained embedding layer of the large language model as a text encoder. For image encoding, the image is first subjected to layer normalization, and then encoded using the block embedding module of the Visual Transformer (ViT). It should be noted that the dimensions of image features and text features must remain consistent during subsequent feature alignment; therefore, this invention does not use the pre-trained ViT model, but rather adjusts the embedding dimensions of ViT accordingly. The extraction process of text embedding and image embedding can be represented as follows: in, , These represent text and image embedding features, respectively. The number of words in the input text. This represents the input dimension of the large language model GLM-4-9B. ,in For the input image size, Set the image block size.

[0028] S3. Construct a cross-modal fusion module, utilize the pre-trained embedding layer weights of the large language model to align the image embedding features to the semantic space of the text embedding features, and obtain aligned image features; then fuse the aligned image features with the text embedding features to obtain fused features; To establish a close connection between text and images, enabling large language models to understand image content more deeply and fully utilize their pre-trained background knowledge, it is necessary to align text and image modalities. To this end, this invention employs a multi-head cross-attention mechanism to align image features with text features. However, the vocabulary of large language models is enormous and contains many redundant words, making it difficult to align image features with every word in the vocabulary one by one. Therefore, a linear layer is needed to adjust the parameters of the embedding layer of the large language model. Mapping is performed to transform word features into word prototypes. Then, the image features... As a query, word prototype For Key and Value. Specifically, for each header ,in The number of heads in the bullish attention is defined here. , , ,in , This structure achieves two goals: (1) effective cross-modal alignment through semantic tagging prototypes; and (2) computational optimization through dimensionality reduction while preserving contextual information. The specific process is as follows: in , , , For the vocabulary dimension, The dimension is the result of a linear mapping. Each head... Accumulate, and you will get , which represents the aligned image features.

[0029] By using the original text features Aligned image features By piecing them together, we obtain the fusion feature. This modality fusion mechanism utilizes the embedding weights of a large language model to align features, activating the pre-trained background knowledge in the model. This enables the large language model to gain a deeper understanding of the irony detection task and addresses the problem of weak modal connections.

[0030] S4. Input the fused features into the large language model, and fine-tune the large language model using a parameter-efficient fine-tuning method to obtain high-dimensional semantic features; Large language models typically have a large number of parameters and strong generalization capabilities, but they may struggle to fully capture the details and specific requirements of a particular task. Therefore, it is necessary to fine-tune the model to better suit the needs of the current task. This involves considering the pre-training parameters of large language models. Add two low-rank matrices and During training, the original computation process for this module... Freeze the main parameters of the large language model Only for two low-rank matrices and The parameters need to be adjusted, and the specific process is as follows: S5. Construct a classification module based on multimodal attention, perform multimodal self-attention enhancement and progressive dimensionality compression on the high-dimensional semantic features, and output the classification result of irony detection.

[0031] In multimodal irony detection tasks, effective decision-making requires models to establish robust cross-modal interactions while avoiding reliance on noisy conditional generation outputs. To this end, this invention first aligns the visual representation with the text embedding space of a large language model, and then explicitly couples these two modalities for subsequent inference: in , Indicates splicing, This represents the multimodal self-attention operation.

[0032] Large language models possess powerful feature processing capabilities and rich background knowledge, but their high output dimensionality poses a significant challenge to downstream classification tasks. Furthermore, commonly used generation heads may produce errors or irrelevant information. Applying traditional single-layer classification heads directly to such high-dimensional features often fails to fully capture their complex information patterns, leading to poor performance. Simultaneously, the rapid decrease in dimensionality during classification may compromise the model's robustness. To address these challenges, this invention designs a classification module with a 1 / 8 decreasing ratio. This module alternates between RMS normalization layers and linear layers, mitigating overfitting risks and forcing the model to capture only the most salient, task-specific patterns while eliminating noise interference. This process can be represented as: in, As an intermediate variable, This module provides the output for a specific layer. It effectively addresses the challenges posed by the high-dimensional output of large language models, significantly improving classification accuracy and model robustness. Furthermore, while existing technologies typically rely on generation tasks to solve specific problems, this invention transforms them into classification tasks, thereby fundamentally suppressing the generation of erroneous or irrelevant information.

[0033] Example 2: This embodiment provides a multimodal irony detection system based on lexical projection of a large language model and semantic alignment of text and images, including: The data acquisition module acquires the multimodal data to be detected, including text data and image data; The feature encoding module encodes the text data and the image data respectively to obtain text embedding features and image embedding features; The cross-modal fusion module utilizes the pre-trained embedding layer weights of the large language model to align the image embedding features to the semantic space of the text embedding features, obtaining aligned image features; then, the aligned image features are fused with the text embedding features to obtain fused features. The large language model fine-tuning module inputs the fused features into the large language model and uses a parameter-efficient fine-tuning method to fine-tune the large language model to obtain high-dimensional semantic features; The classification output module constructs a classification module based on multimodal attention, performs multimodal self-attention enhancement and progressive dimensionality compression on the high-dimensional semantic features, and outputs the classification result of irony detection.

[0034] The above modules can be deployed on the same device or distributed devices; the division of modules is only a functional logic description and does not limit the specific physical boundaries or implementation order.

[0035] Example 3: An electronic device is provided for running the aforementioned "multimodal irony detection method based on large language model lexical projection and image-text semantic alignment". The electronic device includes: a processor, a memory, and optional communication interfaces / display devices / input devices, etc.; the memory stores a computer program that can run on the processor, and when the processor executes the program, it implements steps S1 to S5 of the method described in Embodiment 1, specifically including but not limited to: S1. Acquire the multimodal data to be detected, wherein the multimodal data includes text data and image data; S2. Encode the text data and the image data respectively to obtain text embedding features and image embedding features; S3. Construct a cross-modal fusion module, utilize the pre-trained embedding layer weights of the large language model to align the image embedding features to the semantic space of the text embedding features, and obtain aligned image features; then fuse the aligned image features with the text embedding features to obtain fused features; S4. Input the fused features into the large language model, and fine-tune the large language model using a parameter-efficient fine-tuning method to obtain high-dimensional semantic features; S5. Construct a classification module based on multimodal attention, perform multimodal self-attention enhancement and progressive dimensionality compression on the high-dimensional semantic features, and output the classification result of irony detection.

[0036] The electronic device hardware can be one of a server, personal computer, workstation, industrial controller, edge computing device, or mobile terminal; the processor can be a general-purpose CPU, GPU, NPU, FPGA, or a combination thereof; the memory can be RAM, ROM, flash memory, or disk array. The device can interact with local / remote data storage (acquiring observation data and outputting inversion results) through a communication interface. The above hardware configuration does not constitute a limitation of the present invention.

[0037] Example 4: A computer-readable storage medium storing a computer program, which, when run on a processor of an electronic device, causes the program to execute the method steps S1 to S5 described in Embodiment 1; the storage medium may be a disk, optical disk, flash memory, solid-state drive, read-only memory, random access memory, or any combination of the above media.

[0038] To verify the effectiveness of this invention, the specific configuration of the experimental environment is shown in Table 1. This invention first uniformly scales all images to 224×224 pixels and divides them into 16×16 image blocks. This block-division strategy preserves local image details and facilitates feature extraction by the model. Simultaneously, to align with the 4096-dimensional features output by the GLM-4-9B model, this invention adjusts the embedding dimension of ViT accordingly to ensure consistency in feature dimensions. Regarding the parameter settings for the LoRA method, this invention sets the rank parameter (rank) to 128, the scaling factor (alpha) to 64, the dropout ratio to 0.01, and fine-tunes the query_key_value and dense modules. For the optimization strategy, this invention uses the Adam optimizer. For the SarcNet dataset, the initial learning rate is set to 2e-5, and a cosine annealing decay strategy is used to gradually decrease the learning rate to 2e-9, with the training process lasting for 5 epochs. For the large MMSD dataset, the training process is only set to 3 epochs, with other settings remaining unchanged. All training and testing tasks in this invention are completed on a single NVIDIA RTX 5090 graphics card.

[0039] Table 1 Experimental Environment To evaluate the model's performance, this embodiment uses accuracy (Acc), precision (Pre), recall (Rec), and F1 score (F1) as evaluation metrics: Where TP represents the number of samples that were predicted to be positive and whose actual labels are also positive. FP represents the number of samples that were predicted to be positive but whose actual labels are negative. TN represents the number of samples that were predicted to be negative and whose actual labels are also negative. FN represents the number of samples that were predicted to be negative but whose actual labels are positive.

[0040] This invention selects two image-based models (ResNet and ViT), five text-based models (Bi-LSTM, TextCNN, BERT, Qwen2.5-7B, and GLM-4-9B), and twelve multimodal models. The multimodal models include: one traditional model, Res-BERT; three attention-based models, namely MMFN, FSICN, and HIAN; and four graph neural network-based models, namely KnowleNet, DIP, and G... 2 SAM and DMMD; one large language model-based method, LLaVA1.5-VIDR; and three closed-source large models employing zero-shot inference: Qwen-VL-Max, GPT-4o, and Claude-sonnet-4. A brief description of each main method is as follows: ResNet: By introducing a residual learning framework, it effectively solves the degradation problem in deep network training. This structure breaks through the limitations of network depth, making efficient training of extremely deep networks possible. ViT: Successfully transfers the Transformer architecture from natural language processing to computer vision. This model divides images into fixed-size image patches and linearly projects them into sequences, abandoning traditional convolutional operations. Utilizing a self-attention mechanism, ViT can capture global contextual dependencies in images, overcoming the limitation of CNNs' small receptive field. Bi-LSTM: By combining forward and backward LSTM hidden layers, it can simultaneously capture past and future information in the sequence context. This structure overcomes the limitation of unidirectional models relying only on historical data and can effectively model long-term dependencies between sequences. TextCNN: Applyes convolutional neural networks to text classification. Its core idea is to use one-dimensional convolutional kernels of different sizes to slide on the text sequence to extract key local semantic features. Then, a max-pooling layer is used to select the most discriminative features, and features of different sizes are fused for classification.

[0041] Res-BERT: Focuses on modeling inconsistencies within and between modalities, suitable for sarcasm detection tasks and achieving satisfactory performance. MMFN: Proposes a multi-granularity information fusion network, utilizing the multi-task CLIP framework to capture multi-granularity cues. Through global-local cross-modal interactive learning, this method effectively combines global context and local features, alleviating the weak correlation problem and significantly improving multimodal sarcasm detection performance. FSICN: Captures sarcasm information through dynamically connected fact inconsistency modules and generates cross-modal graphs for retrieving sentiment relationships between text and images. HIAN: A fusion model based on multimodal hybrid interactive attention, combining image-level words and text, and utilizing bidirectional LSTM to enhance feature extraction. KnowleNet: Based on the ConceptNet knowledge base, it incorporates prior knowledge and quantifies the semantic association between images and text through sample-level and word-level cross-modal semantic similarity detection; it also introduces contrastive learning to optimize the spatial distribution of sarcasm and non-sarcasm samples, enhancing the model's ability to distinguish sarcasm cues. DIP: Employs a dual-branch approach of fact and sentiment to mine sarcasm information. The fact branch employs a channel reweighting strategy to obtain semantic discriminative embeddings and utilizes a Gaussian distribution to model the semantic relevance uncertainty caused by inconsistency. This distribution is generated from the latest data in the memory bank and can adaptively distinguish the semantic similarity differences between satirical and non-satirical samples. The sentiment branch learns cross-modal sentiment information through a Siamese layer, combines polarity values ​​to construct a mini-batch relationship graph, and forms a continuous contrastive loss to obtain sentiment embeddings. 2 SAM proposes a reasoning paradigm based on global graph semantic awareness. This model constructs a fine-grained multimodal graph, utilizes global semantic consistency to retrieve k-nearest neighbor instances as prediction references, and combines label-aware graph contrastive learning to improve feature quality.

[0042] DMMD: Proposes a decoupled multi-granularity multimodal distillation framework. This model decouples representations into a modal general subspace and a modal specific subspace, achieving knowledge transfer through adaptive distillation within the subspace and mutual learning between subspaces. LLaVA1.5-VIDR: Based on a multimodal large language model, proposes a generative multimodal irony detection model consisting of instruction templates and a demonstration retrieval module.

[0043] As shown in Table 2, the comparative experimental results demonstrate that the VTA-Sarc model proposed in this invention achieves an average accuracy improvement of over 2%. Analysis of the experimental results reveals certain limitations of existing baseline models. Specifically, while methods based on pre-trained language models (such as Res-BERT) can detect the correlation between images and text, they lack explicit cross-modal alignment, thus limiting classification accuracy. Attention-based models only use the language model for text feature extraction, failing to fully utilize its pre-trained visual-semantic knowledge; furthermore, the limited model parameter size further restricts performance. Although LLaVA1.5-VIDR employs a multimodal LLaVA model, its performance is still limited by the image encoder. In contrast, VTA-Sarc, through a cross-modal fusion module, explicitly fuses image and text modalities using pre-trained weights of a large language model, achieving task-adaptive image-text alignment and effectively eliminating the aforementioned limitations.

[0044] Table 2 Comparison of experimental results It is worth noting that in plain text mode, GLM-4-9B outperforms the model of this invention. This phenomenon may be due to a certain degree of data leakage in the text modality of the MMSD dataset. On the SarcNet dataset, GLM-4-9B performs comparably to BERT. Meanwhile, this invention finds that the results obtained by zero-shot inference closed-source large language models, represented by GPT-4o, are unsatisfactory. These phenomena indirectly indicate that the problem of errors or irrelevant information in generative models is quite serious, and that multimodal large language models are usually designed for general tasks, performing poorly on fine-grained tasks such as irony detection.

[0045] While graph neural network-based models (such as KnowleNet) establish connections between image and text modalities to some extent, they lack rich background information, resulting in poor performance when processing satirical samples. DIP and FSICN fuse text and image features into a relation matrix before subsequent processing, leading to weak connections within the text and image, making it difficult to effectively handle satirical information with weak intermodal associations. In contrast, the VTA-Sarc model systematically optimizes three dimensions: modal alignment, modal association, and pre-training background knowledge. Through precise modal alignment, it strengthens the association between image and text data, allowing the large language model to treat images and text as a whole, rather than two isolated entities. Simultaneously, leveraging the rich pre-training background knowledge of the large language model enhances its generalization ability and accuracy. The classification process is directly completed by the large language model, thus enabling accurate judgment even for satirical information with weak modal associations. This not only enhances image-text relevance and improves generalization ability but also provides stable detection accuracy for satirical samples with weak modal associations.

[0046] This invention conducts comprehensive ablation experiments on the SarcNet dataset to systematically evaluate the impact of cross-modal fusion, LoRA fine-tuning, and low-rank decomposition classification. The experimental scheme sequentially removes each module while keeping the architecture and parameters of other components unchanged to ensure the reliability of the results. The ablation experiment results are shown in Table 3, where "w / o" indicates the removal of a module (without), and "w / " indicates the addition or retention of the module (with).

[0047] Table 3 shows the ablation experiment results on the SarcNet dataset. (1) Effectiveness of the fusion module. The fusion module is the core component of the method of this invention. This module aligns the visual modality and the text modality based on image features and model embedding weights, effectively realizing the fusion of heterogeneous features, thereby enhancing the ability of the large language model to interpret visual-semantic relationships. This module makes a significant contribution, improving the accuracy by more than 2%. After removing this module, the model treats images and text as two isolated sources of information, weakening its ability to understand the relationships between modalities, thus verifying the effectiveness of the fusion module.

[0048] (2) Effectiveness of LoRA fine-tuning. LoRA fine-tuning introduces only about 86M trainable parameters. This module fine-tunes the large language model by adding two low-rank matrices, resulting in a significant improvement in accuracy of about 6%. After removing this module, the parameters of the large language model cannot be adaptively adjusted for the sarcasm detection task. This result proves that LoRA can effectively adapt the large language model to the sarcasm detection task while maintaining parameter efficiency. It also shows that even with limited computing resources, training the large language model is still necessary for sarcasm detection.

[0049] (3) Effectiveness of the classification module. The classification module adds two low-rank matrices on top of the traditional single-layer classification head. Removing this module causes the model performance to drop by about 1%. This shows that although the inherent representation of large language models is rich, specialized architectural components are still needed to fully explore their discriminative potential in irony detection.

[0050] To further explore the differences between the method of this invention and other open-source models, this embodiment selects three representative cases for analysis, such as... Figure 2 As shown in the figure, examples marked "True" in red are positive (ironic), and examples marked "False" in blue are negative (not ironic).

[0051] exist Figure 2In (a), the image shows a woman smiling outdoors with the caption: “It’s cloudy outside, but I’m not happy, Wednesday, folks!” This is a non-sarcasm example. The caption clearly conveys negative emotion, and the positive visual content only creates a slight cross-modal contrast, not constituting a semantic irony reversal. Nevertheless, the DIP model still misclassified this sample as sarcastic, likely because it over-relys on surface lexical cues such as “cloudy” and “not happy.” In contrast, the model of this invention correctly predicted the absence of sarcasm, demonstrating stronger semantic and pragmatic sensitivity, indicating its superior ability to avoid keyword-triggered misidentifications and to reconstruct the speaker’s true intentions.

[0052] exist Figure 2 In (b), the image depicts an interview scene where the interviewer says to the interviewee, "Tell me something you can do that ChatGPT can't." The image clearly contains irony, implying the obstacles individuals face in their job search after ChatGPT's release. The text simply states, "Technical interviews are like this... #Jobs #Hiring," without directly expressing irony. The DIP model incorrectly classifies this case as non-ironic, due to its insufficient understanding of the image's content. In contrast, the method of this invention, through a carefully designed modal alignment mechanism, enables the large language model to successfully identify the deeper meaning of ChatGPT's impact on job hunting. This case demonstrates the superiority of this invention in modal fusion.

[0053] exist Figure 2 In (c), the image depicts extremely slow internet speeds, accompanied by a sentimental message: “Dear <User>, please never change. You are perfect in every way.” The irony here is entirely implicit, stemming from the stark inconsistency between the positive textual assessment and the disheartening visual evidence. The DIP model fails to effectively reconcile these two modalities. In contrast, the fusion module proposed in this invention facilitates a tighter semantic coupling between visual and textual representations, enabling the model to detect such subtle mismatches and correctly infer the satirical intent. This example highlights the advantages of this architecture in handling satirical samples that rely on fine-grained text-image integration.

[0054] To further evaluate the effectiveness of the fusion module proposed in this invention, the attention map between text tokens and their aligned visual features was visualized. Figure 3 Three representative visualization examples are presented. In these heatmaps, the color scale from blue to red indicates that, guided by textual semantics, the model's attention to the corresponding visual region gradually increases.

[0055] The first example depicts a gloomy, foggy roadside scene with the caption "What a beautiful day!!". A heatmap shows the spatial distribution of attention weights. After introducing the cross-modal fusion module proposed in this invention, attention is significantly more focused on the foggy sky and road surface areas. This pattern indicates that the mechanism proposed in this invention strengthens the alignment between salient ironic cues and their supporting visual evidence, enabling the model to better capture subtle irony evoked by the contrast between scene and text.

[0056] The second example shows a takeout order where the fries are noticeably under-served, accompanied by the exaggerated caption, "<User> really filled my takeout order with fries." The heatmap shows that the model primarily focuses on the fries section within the container. This observation demonstrates that the cross-modal fusion module can accurately locate the visual referents supporting satirical utterances, thus associating satirical semantics with corresponding visual entities and supporting fine-grained satirical understanding.

[0057] The third example includes a weather radar map showing a severe storm, captioned "Perfect weather for harvesting wheat." The irony here stems from the stark contrast between the dangerous weather conditions and the positive assessment in the text. The enhanced heatmap highlights the areas in the radar map corresponding to the storm structure. This further demonstrates that the fusion module can effectively associate satirical text with semantically relevant visual features, thereby enabling the model to infer satirical intent from contextual visual information.

[0058] Overall, the visualization results provided strong evidence that the fusion module proposed in this invention not only improves text-image semantic alignment but also models cross-modal interactions more effectively, providing intuitive support for the interpretability and effectiveness of this method.

[0059] To obtain the most suitable parameter settings, this invention conducted experiments on the hyperparameters of LoRA on the SarcNet dataset. LoRA includes two hyperparameters. and .in This represents the dimensions of the two low-rank matrices used for additional fine-tuning. This is a scaling factor that affects the magnitude of LoRA updates. It is set empirically... Set as Half of the time, while relatively improving training efficiency, avoids overfitting. For different... The experimental results of the parameters are as follows Figure 4 As shown, it can be seen that when Setting the matrix dimension to 64 resulted in optimal model performance; further increasing the matrix dimension actually led to a performance decrease. Due to the limited amount of training data, increasing the matrix dimension caused the model to learn noise from the training data, resulting in overfitting. Furthermore, according to the Scaling Laws, without sufficient data support, simply increasing the number of parameters will only lead to a decrease in generalization performance.

[0060] Those skilled in the art will understand that the modules or steps described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, which can then be stored in a storage device for execution by a computer device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. This disclosure is not limited to any particular combination of hardware and software.

[0061] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0062] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for detecting sarcasm in multi-modal text based on large language model vocabulary projection semantic alignment, characterized in that, Includes the following steps: Acquire the multimodal data to be detected, wherein the multimodal data includes text data and image data; The text data and the image data are encoded respectively to obtain text embedding features and image embedding features; A cross-modal fusion module is constructed, which utilizes the pre-trained embedding layer weights of a large language model to align the image embedding features to the semantic space of the text embedding features, thereby obtaining aligned image features. The aligned image features are then fused with the text embedding features to obtain fused features; The fused features are input into a large language model, and the large language model is fine-tuned using a parameter-efficient fine-tuning method to obtain high-dimensional semantic features; A classification module based on multimodal attention is constructed to perform multimodal self-attention enhancement and progressive dimensionality compression on the high-dimensional semantic features, and output the classification result of irony detection.

2. The method of claim 1, wherein the method is characterized in that, By utilizing the pre-trained embedding layer weights of a large language model, the image embedding features are aligned to the semantic space of the text embedding features. Specifically, this includes: passing the embedding layer weight matrix of the large language model through a linear layer. Mapping to low-dimensional word prototypes ,in, For the vocabulary dimension, Let be the dimension after linear mapping, and ; hidden_layer_size for the model; with a multi-head cross-attention mechanism, to the image embedding features as a query, to the word prototypes as key and value, resulting in aligned image features: ; wherein P is the number of image patches, is the query matrix of the i-th attention head, is the query matrix of the i-th attention head, The key matrix of i attention heads Let be the value matrix of the i-th attention head. Let be the dimension of the i-th attention head.

3. The method of claim 2, wherein, in the multi-head cross-attention mechanism, for each attention head Definition: ， ， ; wherein, , , Let H be the image embedding feature matrix, and H be the total number of attention heads in the multi-head attention mechanism. The linear transformation weight matrix used by the i-th attention head to generate the query matrix. The linear transformation weight matrix used by the i-th attention head to generate the key matrix. Let be the linear transformation weight matrix used by the i-th attention head to generate the value matrix; The dimension of the weight matrix; output of each head aligned image features after stitching .

4. The multimodal irony detection method based on large language model lexical projection and image-text semantic alignment according to claim 1, wherein the fusion feature method is as follows: embedding the text features... The aligned image features By concatenating along the feature dimensions, the fused features are obtained. , where n is the number of words.

5. The multimodal irony detection method based on image-text semantic alignment using lexical projection of a large language model according to claim 1, wherein the parameter efficient fine-tuning method is a low-rank adaptive method, that is: for the pre-trained weight matrix of the large language model Introduce two low-rank matrices and The forward propagation process was modified to Freeze during training Only update parameters and ,in This represents the pre-trained original weight matrix in the large language model, where d is the dimension of the output features (i.e., the number of rows in the weight matrix) and K is the dimension of the input features (i.e., the number of columns in the weight matrix). Let r be the feature vector input to this layer, r be the rank of the low-rank decomposition, and h be the final output of this layer.

6. The multimodal irony detection method based on large language model lexical projection and image-text semantic alignment according to claim 1, wherein the classification module based on multimodal attention is implemented as follows: First, multimodal self-attention operations are performed on the high-dimensional semantic features to enhance the interaction between the text and image modalities; Then, following a preset decreasing factor, the feature dimension is gradually reduced by passing through a linear layer and a normalization layer in sequence. Finally, the fully connected layer maps the result to the binary classification output.

7. The image-text semantic alignment multimodal irony detection method based on large language model lexical projection according to claim 1, when encoding the image data, uses the block embedding module of the visual transformer ViT and performs layer normalization processing on the image.

8. A system for detecting sarcasm in multi-modal text based on large language model vocabulary projection semantic alignment, characterized in that, include: The data acquisition module acquires the multimodal data to be detected, including text data and image data; The feature encoding module encodes the text data and the image data respectively to obtain text embedding features and image embedding features; The cross-modal fusion module utilizes the pre-trained embedding layer weights of the large language model to align the image embedding features to the semantic space of the text embedding features, thereby obtaining aligned image features. The aligned image features are then fused with the text embedding features to obtain fused features; The large language model fine-tuning module inputs the fused features into the large language model and uses a parameter-efficient fine-tuning method to fine-tune the large language model to obtain high-dimensional semantic features; The classification output module constructs a classification module based on multimodal attention, performs multimodal self-attention enhancement and progressive dimensionality compression on the high-dimensional semantic features, and outputs the classification result of irony detection.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory to run on the processor, characterized in that, When the processor executes the program, it implements the multimodal irony detection method based on the lexical projection of a large language model for image-text semantic alignment as described in any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the program implements the multimodal irony detection method based on the lexical projection of a large language model for semantic alignment of text and image, as described in any one of claims 1-7.

Images