A Metaphorical Representation Method for Multimodal Large Language Models Based on Edge Detection
By extracting cross-layer features from a multimodal large language model using edge detection technology and performing weighted fusion, and using an MLP classifier for metaphor detection, the problem of insufficient model interpretability in multimodal metaphor detection is solved, and the effective encoding and detection of metaphor knowledge in multimodal large language models is realized.
Patent Information
- Application Number
- CN202510584407.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2045-05-07
AI Technical Summary
Existing technologies lack in-depth exploration of metaphor knowledge encoding in multimodal large language models in multimodal metaphor detection, and the interpretability of multimodal metaphor detection models is insufficient, making it difficult to effectively reveal the metaphorical representations within the model.
Edge detection technology is used to extract cross-layer features from the hidden representations of multimodal large language models. Weighted representations with a unified dimension are generated through projection, pooling and weighted fusion. A multilayer perceptron classifier is used for metaphor classification tasks. During the training phase, the classifier parameters are optimized through the cross-entropy loss function.
By evaluating classifier performance, this study reveals whether multimodal large language models encode metaphorical information, providing a tool for studying the interpretability of metaphorical knowledge within the model. It is applicable to different multimodal large language models and has strong versatility and interpretability.
Smart Images

Figure CN120448919B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of multimodal large language models, metaphor detection, and explainable artificial intelligence, and proposes a metaphor representation method for multimodal large language models based on edge detection. Background Technology
[0002] Metaphor, as a complex linguistic phenomenon, is an indispensable part of human cognition and language expression. Under the framework of Conceptual Metaphor Theory (CMT), metaphor realizes the concretization of abstract concepts by mapping the characteristics of the source domain to the target domain [1]. This mechanism is widely present in human daily language. For example, the metaphor "time is money" emphasizes the preciousness of time by conveying the scarcity and value of money. This cross-domain analogy not only enhances the expressiveness of language, but also promotes the deep understanding of complex concepts by humans. However, the diversity, context dependence, and high demand for background knowledge of metaphor make it a very challenging task in the field of natural language processing (NLP).
[0003] In traditional metaphor detection research, early methods mainly relied on rules and knowledge bases, using predefined linguistic rules to identify metaphors [2]. Although these methods can work effectively in certain specific scenarios, they usually lack adaptability to the diversity of metaphors, especially when facing cross-domain and multimodal contexts. Subsequently, feature engineering-based methods introduced statistical and machine learning models to classify metaphors by extracting features such as semantic similarity and syntactic dependencies [3]. However, these methods are highly dependent on language and task in feature selection, thus limiting their generalization ability.
[0004] In recent years, with the rapid development of deep learning technology, neural network-based methods have gradually replaced traditional methods and become the mainstream direction of metaphor detection. In particular, the rise of pre-trained language models (PLMs) has significantly improved the performance of metaphor detection [4]. For example, language models with Transformer architectures such as BERT [5] [6] capture deeper semantic relationships through contextualized embeddings and have become an important tool for metaphor detection. However, these methods mainly focus on single-modal (text) metaphor detection and lack attention to multimodal metaphors.
[0005] The rise of multimodal learning has opened up new research directions for metaphor detection technology. Joint modeling of multimodal data (such as images and text) enables models to capture complex semantic relationships across modalities [7]. Some studies have attempted to combine images and text for metaphor detection tasks. For example, Xu et al. proposed the C4MMD framework and used the Chain-of-Thought (CoT) method to realize multimodal metaphor detection [8]. However, existing research mainly focuses on performance optimization, while lacking in-depth exploration of the specific mechanisms of multimodal large language models in metaphor knowledge encoding.
[0006] In addition, as the scale of models continues to expand, the interpretability of models has gradually become a research hotspot. In recent years, edge probing technology has been widely used to interpret the semantic representation ability of pre-trained language models [9]. This technology reveals the model's ability to represent specific linguistic phenomena by freezing model parameters and training additional classifiers. For example, Aghazadeh et al. successfully applied edge probing to metaphor research, proving that the intermediate layers of pre-trained models can effectively encode metaphorical knowledge
[10] . In summary, the field of multimodal metaphor detection is still in its early stages of development, and its main challenges include the semantic complexity and diversity of metaphors and the semantic alignment problem between modalities. At the same time, the metaphor encoding of large multimodal language models still needs further exploration. How to use interpretable methods to explore the metaphorical representation inside large multimodal language models has become an urgent problem to be solved.
[0007] [References]
[0008] [1]Lakoff G, Johnson M. Metaphors we live by[M]. University of Chicagopress,2008.
[0009] [2]Fass D.met*:A method for discriminating metonymy and metaphor bycomputer[J].Computational linguistics,1991,17(1):49-90.
[0010] [3]Turney P,Neuman Y,Assaf D,et al.Literal and metaphorical senseidentification through concrete and abstract context[C] / / Proceedings of the2011Conference on Empirical Methods in Natural Language Processing.2011:680-690.
[0011] [4]Song W,Zhou S,Fu R,et al.Verb metaphor detection via contextualrelation learning[C] / / Proceedings of the 59th Annual Meeting of theAssociation for Computational Linguistics and the 11th International JointConference on Natural Language Processing(Volume 1:Long Papers).2021:4240-4251.
[0012] [5]Devlin J.Bert:Pre-training of deep bidirectional transformers forlanguage understanding[J].arXiv preprint arXiv:1810.04805,2018.
[0013] [6]Vaswani A.Attention is all you need[J].Advances in NeuralInformation Processing Systems,2017.
[0014] [7]Liu H,Li C,Wu Q,et al.Visual instruction tuning[J].Advances inneural information processing systems,2024,36.
[0015] [8]Xu Y, Hua Y, Li S, et al. Exploring Chain-of-Thought for Multi-modalMetaphor Detection[C] / / Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers).2024:91-101.
[0016] [9]Tenney I,Xia P,Chen B,et al.What do you learn from context? probingfor sentence structure in contextualized word representations[J].arXivpreprint arXiv:1905.06316,2019.
[0017]
[10] Aghazadeh E, Fayyaz M, Yaghoobzadeh Y. Metaphors in pre-trained language models: Probing and generalization across datasets and languages[J].arXiv preprint arXiv:2203.14139,2022. Summary of the Invention
[0018] This invention proposes a metaphor representation method for multimodal large language models based on edge detection, aiming to explore the metaphor knowledge encoding of multimodal large language models.
[0019] This invention, based on edge detection technology, extracts cross-layer features from the hidden representations of a multimodal large language model. Through projection, pooling, and weighted fusion operations, a weighted representation with unified dimensions is generated. This representation is then used to train a multilayer perceptron (MLP) classifier for binary classification tasks. By observing the performance of the MLP classifier, the invention verifies whether the model internally encodes metaphorical information. In the experiments, this invention designed two input scenarios: images and image-text pairs. While exploring whether the multimodal large language model encodes metaphorical information, comparative analysis reveals the enhancing effect of multiple modalities on metaphorical information encoding.
[0020] This invention is implemented through the following technical solution:
[0021] A metaphor representation method for multimodal large language models based on edge detection, the method is based on a representation extraction module, a pooling module, a hierarchical module, and a classifier, and includes the following steps:
[0022] Offline phase:
[0023] Input data containing metaphorical and non-metaphorical labels into a multimodal large language model to extract hidden representation data h;
[0024] The hidden representation h is linearly projected into a 256-dimensional space according to the following formula to obtain the hidden representation tensor matrix z.
[0025] z = W p h+b p
[0026] in, The weight matrix is for linear projection; Let be the bias vector, and d be the dimension of the original hidden representation;
[0027] The hidden representation tensor matrix p is obtained by pooling and normalizing it using an adaptive attention mechanism. (i) ;
[0028] The hidden weighted representation v is obtained by performing hierarchical weighted summation on the hidden pooling representation matrix using the following formula;
[0029]
[0030] Where, α i Represents the weights of the i-th layer in a multimodal large language model;
[0031] The following formula is used to predict whether the input data contains metaphor labels for hidden weighted representations.
[0032]
[0033] Among them, W c and b c Let be the weight matrix and bias vector, respectively, and σ(·) be the Sigmoid activation function;
[0034] Training phase
[0035] Use the cross-entropy loss function to evaluate the prediction results. The differences between the actual labels and the classifier are analyzed, and the parameters of the classifier are continuously adjusted through the backpropagation algorithm.
[0036] Furthermore, the hidden representation tensor matrix is pooled and normalized using an adaptive attention mechanism to obtain the hidden pooling representation matrix p. (i) The process includes:
[0037] Multi-head self-attention processing is performed on the hidden representation tensor matrix z according to the dimension of the input tensor [B,L,T,D].
[0038] Where: B is the batch size, L is the number of layers, T is the sequence length, and D is the feature dimension;
[0039] Max pooling is performed on the attention-processed representation along the sequence length T:
[0040] p = max(z1, z2, ..., z T )
[0041] The pooled representation is then subjected to L2 normalization according to the following formula to eliminate the influence of feature scale:
[0042]
[0043] Furthermore, the weight α of the i-th layer in the multimodal large language model i for:
[0044]
[0045] Where: β i These are trainable parameters.
[0046] Beneficial effects
[0047] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:
[0048] This invention uses edge detection technology to extract cross-layer features from the hidden representations of a multimodal large language model. It then generates a weighted representation with a unified dimension through projection, pooling, and weighted fusion. This weighted representation is input into a multilayer perceptron (MLP) classifier for training a metaphor classification task. By evaluating the classifier's performance, it is possible to determine whether the model internally encodes metaphorical information.
[0049] In the metaphor representation method for multimodal large language models based on edge detection provided in this invention, by freezing the model parameters, only a small-scale classifier needs to be trained to investigate whether the model encodes metaphorical information. This invention provides theoretical support for metaphor detection research in multimodal large language models by revealing how the model encodes metaphorical knowledge. Furthermore, the method of this invention is applicable to different multimodal large language models, possessing strong versatility, and provides an effective analytical tool for subsequent research on the semantic representation capabilities of multimodal models. Attached Figure Description
[0050] Figure 1 This is an example diagram of the method of the present invention. Detailed Implementation
[0051] To better understand the above-mentioned objectives, features, and advantages of the present invention, the solutions of the present invention will be further described below. The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.
[0052] like Figure 1 As shown, this invention proposes a metaphor representation method for multimodal large language models based on edge detection, comprising the following modules: a representation extraction module, an adaptive attention pooling module, a hierarchical weighted summation module, and a classifier module. The specific implementation of each module is as follows:
[0053] 1. Characterization Extraction Module
[0054] The purpose of the representation extraction module is to extract and project the representations of a multimodal large language model. The specific implementation is as follows:
[0055] 1.1 Extraction and Characterization
[0056] Obtain the model representation h of the input data from the multimodal large language model. This representation contains high-dimensional features of the input data (e.g., images or image-text pairs).
[0057] 1.2 Linear Projection
[0058] Since the representations of different layers in a multimodal large language model have different dimensions, we use a linear transformation to project the representation h to 256 dimensions to align the dimensions. The projection calculation formula is as follows:
[0059] z = W p h+b p
[0060] in, and are the weight matrix and bias vector of the linear projection, respectively, and d is the dimension of the original representation.
[0061] 2. Adaptive Attention Pooling Module
[0062] Self-attention mechanisms can focus on different parts of the input data, capturing long-range dependencies. This is crucial for metaphorical representations, as metaphors often rely on long-range dependencies, such as cross-modal conceptual connections. Pooling operations can extract key information from the representation, generating feature vectors with global semantics. The specific implementation is as follows:
[0063] 2.1 Multi-head self-attention mechanism
[0064] Multi-head self-attention processing is applied to the projected tensor z. The input tensor has dimensions [B,L,T,D], where B is the batch size, L is the number of layers, T is the sequence length, and D is the feature dimension.
[0065] 2.2 Pooling
[0066] Max pooling is performed on the attention-processed representation along the sequence length T:
[0067] p = max(z1, z2, ..., z T )
[0068] 2.3 Normalization Processing
[0069] The pooled representations are then subjected to L2 normalization to eliminate the influence of feature scale:
[0070]
[0071] After L2 normalization, all feature vectors are adjusted to the same order of magnitude, ensuring that features from different input samples are treated fairly during model training and comparison.
[0072] 3. Hierarchical weighted summation module
[0073] The hierarchical weighted summation module aims to fuse feature representations from different levels to capture multi-layered semantic information. The specific implementation is as follows:
[0074] 3.1 Hierarchical Weight Calculation
[0075] The weights α for each level are calculated using the softmax function. i :
[0076]
[0077] By introducing a trainable parameter β i The classifier uses a softmax operation to calculate weights, automatically assigning weights based on the importance of different layers in representing metaphorical information. This process is adaptive, dynamically adjusting according to the characteristics of the training data and the model's learning progress, thereby fully extracting metaphor-related information from each layer.
[0078] 3.2 Weighted Summation
[0079] To integrate information from multiple layers, the pooled representations of each layer are weighted and summed according to the calculated weights to form the final weighted representation vector. This represents the representation of the i-th layer after max pooling and L2 normalization (i.e., p in the previous text). normBy weighted fusion of these representations, the semantic information contained in different levels can be fully utilized. The final weighted representation v is calculated as follows:
[0080]
[0081] This weighted summation method avoids the limitations of simply averaging information from each layer or relying solely on information from a single layer.
[0082] 4. Classifier Module
[0083] The classifier module is used for metaphor detection based on the weighted representations obtained in the previous steps. A multilayer perceptron (MLP) is used as the classifier, with the weighted representation v as its input. The calculation formula is as follows:
[0084]
[0085] Where φ(·) represents the activation function, σ(·) is the sigmoid function, and W c and b c These are parameters of the MLP layer. This is the final prediction result. The activation function φ(·) introduces nonlinearity into the model, enabling it to learn complex nonlinear relationships in the input data. In this invention, after evaluating and comparing the performance of different activation functions, the ReLU activation function was selected. The Sigmoid function is used to map the model's output to probability values between 0 and 1, facilitating binary classification. The classifier predicts whether the input contains metaphorical information based on the binary labels in the corresponding dataset.
[0086] During training, the cross-entropy loss function is used to measure the difference between the model's predictions and the true labels. The parameters of the MLP are continuously adjusted using the backpropagation algorithm to gradually reduce the loss function, thereby optimizing the classifier's performance. The classifier's performance reflects the degree to which the multimodal large language model encodes metaphorical knowledge. If the classifier's predictions match the labels, it indicates that the model has effectively encoded relevant metaphorical information.
[0087] This embodiment selects three datasets widely used in multimodal metaphor detection research as experimental materials: MultiCMET, MultiMET, and MetMEME. Based on the data source, MultiMET is further divided into three subsets: MultiMET_ads, MultiMET_Facebook, and MultiMET_Twitter; MetMEME is divided into two subsets: MetMEME_Chinese and MetMEME_English. Detailed information about the relevant datasets is shown in Table 1.
[0088] During the experiments, we randomly selected 1000 samples from each dataset, including 500 metaphorical data points and 500 non-metaphorical data points, to ensure that the accuracy and Macro-F1 of the fair coin random baseline were 50% in all cases. To ensure the reliability of the experimental results, the datasets were divided into training, validation, and test sets in a ratio of 0.8 / 0.1 / 0.1.
[0089] Table 1. Dataset Information Statistics
[0090] Dataset Metaphorical data volume Non-metaphorical data volume Total data volume MultiCMET 6,306 7,514 13,820 MultiMET_ads 2,537 1,653 4,190 MultiMET_Facebook 1,066 745 1,811 MultiMET_Twitter 2,555 1,943 4,498 MetMEME_Chinese 2,327 3,718 6,045 MetMEME_English 1,114 2,886 4,000
[0091] This embodiment uses two metrics for evaluation: Accuracy and Macro-F1. Accuracy refers to the proportion of correctly classified samples out of the total number of samples, directly reflecting the model's prediction accuracy across all samples. The higher the value, the better the overall prediction accuracy of the model. Macro-F1 is a metric that comprehensively considers recall and precision. It calculates the F1 score for each class and takes the average. The higher the value, the more balanced the model's prediction performance across all classes.
[0092] This embodiment selects three multimodal large language models: LLaVA-1.5-7B, Qwen-VL, and mBLIP-BLOOMZ-7B. These models support image and text input and have been pre-trained on large-scale data. The hidden layer representations of all models are processed through the representation extraction module, adaptive attention pooling module, and hierarchical weighted summation module proposed in this invention, and finally, the metaphor classification task is completed by an MLP classifier. The experiment uses Adam as the optimizer, with an initial learning rate of 5e-5, a batch size of 16, and 100 training epochs.
[0093] Table 2 shows the classifier performance based on the representations trained by various multimodal large language models.
[0094]
[0095]
[0096] The experimental results in Table 2 show that by learning representations of metaphorical images or their combination with textual descriptions, and training a classifier based on these representations, the classifier's accuracy in the metaphor detection task is significantly higher than the fair coin random baseline (50.00). This result confirms that multimodal large language models can effectively encode metaphorical information. Furthermore, comparing experimental results with pure image input and image-text pair input on the same dataset reveals that the classifier trained using both image and text input performs better than the classifier trained using only pure image input. This result indicates that combined inputs from multiple modalities can enrich the metaphorical representations within a multimodal large language model.
[0097] In summary, the proposed edge-detection-based metaphor representation method for multimodal large language models effectively mines the encoding ability of multimodal large language models for metaphorical information through a representation extraction module, an adaptive attention pooling module, a hierarchical weighted summation module, and a classifier module. Experimental results not only verify that multimodal large language models can effectively encode metaphorical information but also show that multimodal combined inputs can enrich the metaphorical representations within the model. This method provides an effective research approach and methodology for the interpretability of multimodal metaphor detection, possessing certain innovation and practicality, and is expected to provide important reference and guidance for subsequent related research and applications.
[0098] Although the present invention has been described above, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many modifications under the guidance of the present invention without departing from the spirit of the present invention, and these modifications are all within the protection scope of the present invention.
Claims
1. A metaphor representation method for multimodal large language models based on edge detection, characterized in that, The method is based on a representation extraction module, a pooling module, a hierarchical module, and a classifier. The input data is an image or an image-text pair, and includes the following steps: Data containing metaphorical and non-metaphorical labels are input into a multimodal large language model to extract hidden representation data. ; For the hidden representation data The hidden representation tensor matrix is obtained by performing a linear projection into the 256-dimensional space using the following formula. in, Bias vector, The dimension of the original hidden representation; The hidden representation tensor matrix is obtained by pooling and normalizing it using an adaptive attention mechanism. ; The hidden weighted representation is obtained by performing a hierarchical weighted summation of the hidden pooling representation matrix using the following formula. ; in, Represents the first in a multimodal large language model Layer weights; The following formula is used to predict whether the input data contains metaphor labels for hidden weighted representations. ; in, and The weight matrix and bias vector are respectively. Sigmoid activation function ; Use the cross-entropy loss function to evaluate the prediction results. The differences between the actual labels and the classifier are analyzed, and the parameters of the classifier are continuously adjusted through the backpropagation algorithm.
2. The metaphor representation method for multimodal large language models based on edge detection according to claim 1, characterized in that, The hidden representation tensor matrix is obtained by pooling and normalizing it using an adaptive attention mechanism. The process includes: Based on the dimension of the input tensor For the hidden representation tensor matrix Perform multi-head self-attention processing; in: For batch size, For the number of floors, For sequence length, For feature dimensions; The representation after attention processing along the sequence length Perform max pooling operation: The characterization of the pooling result is performed according to the following formula. Normalization is applied to eliminate the influence of feature scale: 。 3. The metaphor representation method for multimodal large language models based on edge detection according to claim 1, characterized in that, The multimodal large language model in the first Layer weights for: in: These are trainable parameters.
Citation Information
Patent Citations
Metaphor identification method based on cross-modal multilayer information fusion
CN118278415A
Modal characterization decomposition method for multi-modal metaphor detection
CN118673401A
