A self-verifying method and system for exploring metaphor knowledge in pre-trained language models.

Abstract

This invention belongs to the field of natural language processing (NLP) technology, specifically relating to a self-verifying method and system for exploring simile knowledge in pre-trained language models. Step 1: Construct a multi-level simile recognition dataset; Step 2: Create a multi-level simile recognition task to help PLMs evaluate simile quality; Step 3: Integrate the multi-level simile recognition scores into the simile generation task. This invention addresses the problem that existing pre-trained language models cannot effectively evaluate the quality of similes, thus hindering the exploration of simile knowledge.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a self-verifying method and system for exploring metaphorical knowledge in pre-trained language models. Background Technology

[0002] Simile is a common linguistic phenomenon in everyday communication, playing a vital role in making speech more vivid, interesting, and easy to understand (Vlad Niculae and Cristian Danescu-Niculescu-Mizil. Brighter than gold: Figurative language in user-generated comparisons. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014; Jiayi Zhang, Zhi Cui, Xiaoqiang Xia, Yalong Guo, Yanran Li, Chen Wei, and Jianwei Cui. Writing polishment with simile: Task, dataset, and Aneural approach. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021.). Simile compares two things from different categories (called the subject and the metaphor) by sharing attributes (Anthony M. Paul. Figurative language. In Philosophy & Rhetoric, 1970.). The subject and the metaphor are usually connected by comparative words such as "like" or "as". For example, "The man is as strong as a bull" is a simile, where the subject is "The man", the vehicle is "a bull", the comparative word is "as...as", and the shared property is "strong". In contrast, "The girl looks like her mother" is not a simile because it compares two things in the same category (comparing the appearance of two people).This linguistic phenomenon is often referred to as literal in simile research (Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. Metaphors in pre-trained language models: Probing and generalization across datasets and languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022; Rowan Hall Maudslay and Simone Teufel. Metaphorical polysemy detection: Conventional metaphor meets word sense disambiguation. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022). In this paper, literal is translated as non-literal simile.

[0003] There are various types of metaphor tasks in natural language processing research. Different types of metaphor tasks require models to have different types of metaphor knowledge. For example, the metaphor recognition (SR) task is a binary classification task (Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. Metaphor detection with cross-lingual model transfer. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014), which requires the model to determine whether a text sequence (e.g., triples or sentences) is a metaphor. The Simile Interpretation (SI) task requires models to either generate interpretive text for similes (Yuri Bizzoni and Shalom Lappin. 2018. Predicting human metaphor paraphrase judgments with deep neural networks. In Proceedings of the Workshop on Figurative Language Processing, Fig-Lang@NAACL-HLT 2018) or infer shared properties between the ontology and the metaphor (Qianyu He, Sijie Cheng, Zhixu Li, Rui Xie, and Yanghua Xiao. Can pre-trained language models interpret similes as smart as human? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL2022).The simile generation (SG) task requires the model to generate either a missing metaphor (Wei Song, Jingjin Guo, Ruiji Fu, Ting Liu, and Lizhen Liu. A knowledge graph embedding approach for metaphor processing. IEEE ACM Trans. Audio Speech Lang. Process. 2021; Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Xiangpeng Wei, Zhengyuan Liu, and JunXie. Fantastic expressions and where to find them: Chinese simile generation with multiple constraints. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023) or a simile sentence (Yucheng Li, Chenghua Lin, and Frank Guerin. Cm-gen: A neural framework for Chinese metaphor generation with explicit context modelling. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022).

[0004] In recent years, pre-trained language models (PLMs) have achieved great success in the field of natural language processing because they learn general knowledge from large corpora (Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020). In simile research, considerable effort has been made to explore the extraction of simile knowledge from PLMs to solve SR / SI / SG tasks (Weijie Chen, Yongzhu Chang, Rongsheng Zhang, Jiashu Pu, Guandan Chen, Le Zhang, Yadong Xi, Yijiang Chen, and Chang Su. Probing simile knowledge from pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022).For example, a widely adopted approach is to fine-tune PLMs to suit specific simile tasks (Tuhin Chakrabarty, Smaranda Muresan, and Nanyun Peng. Generating similes effortlessly like a pro: A style transfer approach for simile generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020). However, previous work has typically explored only one type of simile knowledge to address specific simile tasks, which does not fully utilize the capabilities of PLMs, as previous research has also shown that PLMs possess multiple types of simile knowledge (Qianyu He, Xintao Wang, Jiaqing Liang, and Yanghua Xiao. MAPS-KB: A million-scale probabilistic simile knowledge base. In Third-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023).

[0005] This invention investigates how to utilize multiple types of simile knowledge to complete simile tasks. Specifically, when using PLMs to perform simile tasks (SI / SG), this invention aims to enable PLMs to utilize other indirect simile knowledge (SR) in addition to simile knowledge directly related to the task. Indirect simile knowledge can serve as a self-validating perspective to improve the performance of the target task, thereby fully utilizing the multiple types of simile knowledge in PLMs. However, gaps exist between different types of simile knowledge. For example, SR tasks require PLMs to distinguish between similes and non-similes, while SI / SG tasks require PLMs to generate missing simile components. SR and SI / SG tasks differ in task format and training objects. Directly utilizing simile recognition knowledge to assist SI / SG tasks is difficult. To address this issue, this invention proposes a novel task called multi-level simile recognition (MLSR) to align different types of simile knowledge. The MLSR task extends the SR task from binary classification to multi-class classification. After training with MLSR, PLMs can not only distinguish between similes and non-similes but also assign quality scores to given similes. This quality score can then serve as a self-validation mechanism to evaluate the generated results of the SI / SG task and reorganize the results based on the evaluation. In this way, the present invention integrates different types of metaphor knowledge from PLMs and makes better use of them. Experiments on multiple metaphor datasets demonstrate that the method of the present invention achieves new state-of-the-art performance. The present invention has also been tested on different PLMs, showing that the proposed self-validation method is applicable to different types of PLMs and can be applied to larger-scale language models. Summary of the Invention

[0006] This invention provides a self-verifying method for exploring simile knowledge in pre-trained language models, addressing the problem that existing pre-trained language models cannot effectively evaluate the quality of similes, thus hindering the exploration of simile knowledge.

[0007] This invention provides a self-verifying system for exploring metaphorical knowledge in a pre-trained language model, and a method for realizing a self-verifying approach to exploring metaphorical knowledge in a pre-trained language model.

[0008] The present invention also provides a computer device.

[0009] The present invention also provides a computer-readable storage medium.

[0010] The invention is achieved through the following technical solution:

[0011] A self-verifying method for exploring metaphorical knowledge in a pre-trained language model, the method comprising the following steps:

[0012] Step 1: Construct a multi-level metaphor recognition dataset;

[0013] Step 2: Create a multi-level simile recognition task to help PLMs evaluate simile quality;

[0014] Step 3: Integrate the multi-level metaphor recognition scores into the metaphor generation task.

[0015] Furthermore, step 1 specifically involves constructing metaphor data with multiple metaphor levels; and using the metaphor data to train PLMs to assign quality scores to each level.

[0016] Furthermore, four levels of metaphor with different qualities are defined;

[0017] The first level is a correct simile, in which there are three healthy simile relationships between the components of the simile; the remaining levels are constructed by replacing the components of the simile, resulting in varying degrees of damage to the simile relationships.

[0018] The second level has two replacement methods. The first replacement method is to replace the metaphor with its superordinate word.

[0019] The second substitution method is to replace the metaphor with a synonym of the subject;

[0020] The third level replaces the metaphor or shared attribute according to the replacement rules;

[0021] The fourth level uses replacement rules to replace all shared attributes and metaphors.

[0022] Furthermore, the replacement rules firstly replace words with words of the same part of speech as the original words, that is, shared attributes are replaced with adjectives, while the subject / metaphor is replaced with nouns / phrases / sentences that can serve as verb subjects / objects;

[0023] Secondly, we can use the relationships in ConceptNet to select more suitable words;

[0024] In the second level, the "IsA" relation in ConceptNet is used to find the hypernym of the subject / metaphor; in the fourth level, when replacing the metaphor, nouns / phrases that do not share attributes with the metaphor are selected; when replacing attributes, adjectives that are not related to the subject and metaphor are selected.

[0025] When a simile component contains more than one word, use the first noun or adjective in the component for replacement;

[0026] When no metaphor component is available in ConceptNet, or a suitable replacement cannot be found, only the first rule is followed, and words are randomly selected from ConceptNet.

[0027] Furthermore, step 2 specifically involves training the PLM using a multi-level contrastive learning method; for each metaphor level, a reference score between 0 and 1 is assigned (1.000, 0.667, 0.333, 0.000); during training, the MLSR model learns to predict the score of the input sequence; the training objective is to minimize the distance between the predicted score and the reference score; assuming a linear relationship between the reference quality score and the metaphor level, the model automatically learns the quality scores for different levels.

[0028] The model structure of MLSR when using BERT as the backbone; the first output position is used as the sequence representation; when applying the method to other PLMs with decoder structures, the first decoded vector is used as the input representation to compute the MLSR loss; the input sequence is represented as , the set of input sequences is represented as D, and a sentence in D is represented as h. D h D A multilayer perceptron is used to obtain the predicted quality score S of the input sequence D. D :

[0029] S D =σ(W2·μ(W1·h) D +b1)+b2),

[0030] Using the Sigmoid and tanh functions respectively, each metaphor level is represented as D. i (i∈{1, 2, 3, 4}); then, the MLSR model is trained using separation loss and compactness loss.

[0031] Furthermore, the separation loss distinguishes different metaphor levels by differentiating the quality scores representing different metaphor levels; for each metaphor level i, a center score is first calculated. in This is the quality score at this level, K. i D in the training batch i The number of examples; the separation loss between different metaphor levels is:

[0032]

[0033] Where λ = 0.333 is the lower bound of the distance between the two center scores, and ω = lj is the weight used to amplify the lower bound based on the metaphor quality gap;

[0034] Compactness loss compresses examples within the same metaphorical hierarchy together to avoid outliers at each level;

[0035] Specifically, for k∈{1, 2, ..., K} i For each example in}, its metaphorical quality score Forced to be associated with the corresponding center score Closer, as shown below:

[0036]

[0037] Where γ is the upper bound of the distance between the center point of a replacement level and the scores within that level; γ allows the model to automatically adjust the quality score of each level within a certain range; the final loss of MLSR is:

[0038]

[0039] Where N is the number of training samples; after training, the MLSR model can assign a metaphor quality score to the input sequence.

[0040] Furthermore, step 3 specifically involves the following steps: when using BERT as the backbone network, the SI and SG tasks are identical to the MLM tasks in BERT; BERT learns to recover the masked attributes or metaphors; when using other PLMs with decoder structures for the SI / SG tasks, the PLMs are provided with a task indication and a masked sequence, and are required to generate the missing components; the loss function is:

[0041]

[0042] in, The t-th word representing the hidden attribute / metaphor.

[0043] Train a single PLM to handle all three metaphor tasks: MLSR, SI, and SG; the loss function is... In SI / SG testing, following previous work, PLMs select missing attributes or metaphors from the candidate pool;

[0044] When experimenting with BERT, the masked word prediction head of BERT is used to calculate the probability of each candidate; the candidate with the highest probability is selected as the final choice; when using other large PLMs with decoder structures for SI / SG tasks, an instruction is provided and the PLMs are asked to generate the missing components; the probability of each candidate is the generation probability.

[0045] To utilize MLSR results for self-validation, each candidate is concatenated to the input sequence as a new input; the model can then compute the MLSR score for that candidate; the final score S for each candidate is... final It is the MLSR score S mlsr and SI / SG score S si / sg Weighted sum:

[0046] S final=β*S mlsr +(1-β)*S si / sg

[0047] Where β is a hyperparameter.

[0048] A self-verifying system for exploring metaphorical knowledge in a pre-trained language model, the system employing the self-verifying method for exploring metaphorical knowledge in a pre-trained language model as described above, the system comprising:

[0049] The dataset construction module builds a multi-level metaphor recognition dataset.

[0050] The simile recognition creation module creates multi-level simile recognition tasks to help PLMs evaluate simile quality; the MLSR score serves as a self-validation mechanism to align simile knowledge used in different types of simile tasks.

[0051] The simile generation module integrates multi-level simile recognition scores into the simile generation task.

[0052] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0053] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0054] The beneficial effects of this invention are:

[0055] This invention extends the traditional simile recognition (SR) task from binary classification to multi-class classification. After training with MLSR, PLMs can not only distinguish between similes and non-similes, but also assign quality scores to given similes.

[0056] This invention utilizes MLSR scores as a self-validating mechanism to evaluate results in a metaphor interpretation / generation (SI / SG) task and reorganizes the results based on the evaluation. In this way, the invention integrates and better utilizes different types of metaphor knowledge from PLMs.

[0057] This invention significantly improves the accuracy of metaphor knowledge in pre-trained language models. Experiments on multiple metaphor datasets demonstrate that the method of this invention achieves new state-of-the-art performance (more than 5% improvement in accuracy compared to existing methods).

[0058] This invention also tested the method with different PLMs, and the results showed that the self-validating method is applicable to different types of PLMs, such as the T5 model (Exploring the limits of transfer learning with a unified text-to-text transformer), and can be applied to language models with larger scales. Attached Figure Description

[0059] Figure 1 This is a schematic diagram illustrating four metaphor levels with different metaphor qualities as defined in this invention.

[0060] Figure 2 This is a flowchart of the training and validation process when using BERT as the backbone network in this invention.

[0061] Figure 3 This is a flowchart of the method of the present invention. Detailed Implementation

[0062] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0063] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0064] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0065] The following is in conjunction with the appendix to this application specification. Figure 1-3 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0066] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0067] Example 1

[0068] This embodiment provides a self-verifying method for exploring metaphorical knowledge in a pre-trained language model, the method comprising the following steps.

[0069] Step 1: Construct a multi-level metaphor recognition dataset;

[0070] Step 2: Create a multilevel metaphor recognition (MLSR) task to help PLMs evaluate metaphor quality; whereby the MLSR score serves as a self-validation mechanism to align metaphor knowledge used in different types of metaphor tasks.

[0071] Step 3: Integrate the multi-level metaphor recognition (SI) score into the metaphor generation (SG) task.

[0072] Furthermore, step 1 specifically involves constructing metaphor data with multiple metaphor levels; and using the metaphor data to train PLMs to assign quality scores to each level.

[0073] Furthermore, in Figure 1 In the text, four levels of metaphor with different qualities are defined;

[0074] The first level is a correct simile, in which there are three healthy simile relationships between the components of the simile (as shown in the fourth column); the remaining levels are constructed by replacing the components of the simile and resulting in varying degrees of damage to the simile relationships.

[0075] The second level has two substitution methods. The first method involves replacing the metaphor with its superordinate noun; for example, "The athlete runs as fast as a cheetah" is changed at this level to "The athlete runs as fast as an animal." The resulting sentence is still a simile, comparing two different things that share a property. However, it is less precise than the original sentence because not all animals "run fast." While understanding the sentence involves comparing "the athlete" to "an animal that can run fast," "a cheetah" is more specific than "an animal," making the simile more vivid and easier to understand. Figure 1As shown in the fourth column, the relationship between shared attributes and metaphors may be altered and impaired as a result;

[0076] The second substitution method involves replacing the metaphor with a synonym of the subject; for example, "The athlete runs as fast as a cheetah." at this level is changed to "The athlete runs as fast as a player." A simile requires comparing two things in different categories. "The athlete" and "a player" belong to the same category, thus breaking the simile relationship between the subject and the metaphor.

[0077] The third level replaces the metaphor or shared attribute according to the replacement rules; in Figure 1 In the sentence, "The athlete runs as fast as a cheetah," is replaced with "The athlete runs as fast as air" / "The athlete runs as green as a cheetah." When replacing the metaphor / shared attribute, the shared attribute "fast" still belongs to the subject "athlete," but is unrelated to the metaphor "air." When replacing the shared attribute, the sentence still compares two different things ("athlete" and "cheetah"), but the shared attribute "green" is incorrect. For example... Figure 1 As shown in the fourth column, in the third level, two of the three metaphorical relationships may be impaired.

[0078] The fourth level uses replacement rules to replace all shared attributes and metaphors. Figure 1 In the original text, "The athlete runs as fast as a cheetah" was changed to "The athlete runs as green as black eyes." Not only is "green" not a shared attribute between "athlete" and "black eyes," but "athlete" and "black eyes" also lack comparable attributes. At this level, all three metaphorical relationships are potentially compromised.

[0079] Furthermore, the replacement rules for the third and fourth layers are as follows: First, words with the same part of speech as the original words are used for replacement, that is, shared attributes are replaced with adjectives, while the subject / metaphor is replaced with nouns / phrases / sentences that can serve as verb subjects / objects;

[0080] Secondly, the relationships in ConceptNet (Robyn Speer, Joshua Chin, and Catherine Havasi. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017.) are used to select more appropriate words;

[0081] In the second level, the "IsA" relation in ConceptNet is used to find the hypernym of the subject / metaphor; in the fourth level, when replacing the metaphor, nouns / phrases that do not share attributes with the metaphor are selected; when replacing attributes, adjectives that are not related to the subject and metaphor are selected.

[0082] When a simile component contains more than one word, use the first noun or adjective in the component for replacement;

[0083] When no metaphor component is available in ConceptNet, or when a suitable replacement cannot be found, only the first rule (i.e., use a word with the same part of speech as the original word for replacement) is followed, and a word is randomly selected from ConceptNet.

[0084] Furthermore, step 2 specifically involves the present invention employing a multi-level contrastive learning method to train the PLM; for Figure 1 Each metaphor level in the model is assigned a reference score between 0 and 1 (1.000, 0.667, 0.333, 0.000) to reflect the quality of the metaphor. During training, the MLSR model learns to predict the score of the input sequence. The training objective is to minimize the distance between the predicted score and the reference score. It is important to note that a linear relationship is assumed between the reference quality score and the metaphor level. Figure 1 The relationship between metaphorical elements in the model is disrupted; this setting facilitates model training.

[0085] Figure 2 (a) shows the model structure of MLSR when using BERT as the backbone; the first output position (a special marker before the input sequence) is shown. <cls>The first decoded vector is used as the sequence representation to compute the multi-level SR loss. When the method is applied to other PLMs with decoder structures (Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020), the first decoded vector is used as the input representation to compute the MLSR loss; the input sequence is represented as , the set of input sequences is represented as D, and a sentence in D is represented as h. D h D A multilayer perceptron (MLP) is used to obtain the predicted quality score S of the input sequence D. D :

[0086] S D =σ(W2· μ (W1·h D +b1)+b2),

[0087] Use the Sigmoid function and the tanh function respectively; Figure 1 Each level of metaphor is represented by D. i (i∈{1, 2, 3, 4}); then, the MLSR model is trained using separation loss and compactness loss.

[0088] Furthermore, the separation loss distinguishes different metaphor levels by differentiating the quality scores representing different metaphor levels; for each metaphor level i, a center score is first calculated. in This is the quality score at this level, K. i D in the training batch i The number of examples; the separation loss between different metaphor levels is:

[0089]

[0090] Where λ = 0.333 is the lower bound of the distance between the two center scores, and ω = lj is the weight used to amplify the lower bound based on the metaphor quality gap;

[0091] The compactness loss compresses examples within the same metaphor level together, which acts as a normalization mechanism to avoid outliers at each level.

[0092] Specifically, for k∈{1, 2, ..., K} i For each example in}, its metaphorical quality score Forced to be associated with the corresponding center score Closer, as shown below:

[0093]

[0094] Where γ is the upper bound of the distance between the center point of a replacement level and the scores within that level; γ allows the model to automatically adjust the quality score of each level within a certain range; the final loss of MLSR is:

[0095]

[0096] Where N is the number of training samples; after training, the MLSR model can assign a metaphor quality score to the input sequence.

[0097] Furthermore, step 3 specifically involves, as follows: Figure 2 As shown in (b), when using BERT as the backbone network, the SI and SG tasks are the same as the MLM tasks in BERT; where BERT learns to recover the masked attributes or metaphors; when using other PLMs with decoder structures (e.g., the T5 model) for the SI / SG tasks, the PLMs are provided with a task instruction and the masked sequence, and are required to generate the missing components; the loss function is:

[0098]

[0099] in, The t-th word representing the hidden attribute / metaphor (out of a total of r words, r≥1).

[0100] Train a single PLM to handle all three metaphor tasks: MLSR, SI, and SG; the loss function is... In SI / SG testing, following previous work, PLMs select missing attributes or metaphors from the candidate pool;

[0101] like Figure 2 As shown in (c), when experimenting with BERT, the masked word prediction head of BERT is used to calculate the probability of each candidate; the candidate with the highest probability will be selected as the final choice; when using other large PLMs with decoder structures for SI / SG tasks, an instruction is provided and the PLMs are asked to generate the missing components; the probability of each candidate is the generation probability.

[0102] To utilize MLSR results for self-validation, each candidate is concatenated to the input sequence (to the masking location) as a new input; the model can then compute the MLSR score for that candidate; the final score S for each candidate is... final It is the MLSR score S mlsr and SI / SG score S si / sg Weighted sum:

[0103] S final =β*S mlsr +(1-β)*S si / sg

[0104] Where β is a hyperparameter.

[0105] Specifically, regarding the test datasets, the Multi-Select Probe Dataset (MCP) (Qianyu He, Sijie Cheng, Zhixu Li, Rui Xie, and Yanghua Xiao. Can pre-trained language models interpret similes as smart as human? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022) is proposed for the SI task. It includes two test sets, named the General Corpus and the Quiz. The Multilingual Analogy Dialogue Dataset (MSD) (Longxuan Ma, Weinan Zhang, Shuhan Zhou, Churui Sun, Changxin Ke, and Ting Liu. 2023. I run as fast as a rabbit, can you? A multilingual simile dialogues datasets. In Findings of the Association for Computational Linguistics: ACL 2023) aims to study analogies in dialogue. This invention uses only the English data from the MSD. Both the SI and SG test sets are multi-select tasks, each with 450 examples. Both MCP and MSD annotate all analogy components. Therefore, they are suitable for constructing multi-level analogy data. The differences between MCP and MSD are: 1) Analogies in MSD exist within 3-turn dialogues, with dialogue lengths significantly longer than sentences in MCP; 2) The subjects and metaphors in MSD contain various formats, such as verb phrases and sentences. Table 1 shows the statistics of the dataset. For the MLSR task, this invention constructs the same number of examples for each analogy level (the same as the training set) and randomly splits all data in a 9:1 ratio as the training / validation set.

[0106] Table 1

[0107]

[0108] Regarding the evaluation criteria, this invention uses Hit@1 to measure the accuracy of multiple selections. For MLSR, this invention reports the accuracy for each analogy level. Table 2 shows the experimental results.

[0109] It can be seen that the MLSR task of this invention helps PLMs to assign reasonable quality scores to different metaphor levels defined in this invention, and the performance does not degrade after joint training.

[0110] Table 2

[0111]

[0112] The experiments related to the metaphor generation task used Hit@1 as the evaluation metric. Table 3 shows the performance of the method of this invention on BERT and T5 models on MCP and MSD data.

[0113] Table 3

[0114]

[0115] As can be seen, the self-verification method of this invention helps BERT-base and BERT-large models achieve higher performance improvements. The BERT-base model outperforms the powerful BERT-large(mlm+ke) model in both SI and SG. The BERT-large model improves performance by 4.59% / 5.78% / 5.99% on SI(MCP) / SI(MSD) / SG(MSD), respectively. The accuracy on the MCP SI test set is much higher than that on the MSD SI test set, indicating that the analogy in dialogue (MSD) task is more difficult than the analogy in single sentence (MCP) task. In the second group, T5-3B also exhibits a similar trend to the first group. When using the self-verification method of this invention, T5-3B shows consistent improvement on all three test sets. The results show that the method of this invention can explore more accurate metaphorical knowledge from autoencoder PLMs (e.g., BERT) and sequence-to-sequence masked PLMs (e.g., T5).

[0116] This invention proposes a self-verifying method for exploring simile knowledge from PLMs. A multi-level analogy recognition (MLSR) task is designed to train PLMs to evaluate analogy quality. The MLSR score aligns the analogy recognition knowledge with the analogy generation knowledge and is used to assist SI / SG tasks. This technique can be used to pre-train language models, enabling them to better explore relevant knowledge in sentences containing similes. In terms of implementation, because deep learning techniques are used, a corresponding deep learning framework is required: the relevant experiments are based on the open-source framework PyTorch. If needed, other frameworks can be used, such as the equally open-source TensorFlow, or PadlePadle used internally by enterprises. Similarly, although the embodiments only use English, due to the universality of simile definitions, especially important elements of similes (ontology, shared attributes, metaphor), across different languages, this invention is also applicable to other languages.

[0117] Example 2

[0118] This embodiment provides a self-verifying system for exploring metaphorical knowledge in a pre-trained language model, using the self-verifying method for exploring metaphorical knowledge in a pre-trained language model as described in Embodiment 1. The system includes a dataset construction module, a metaphor recognition and creation module, and a metaphor generation module.

[0119] The dataset construction module uses a self-verifying method to construct a multi-level metaphor recognition dataset.

[0120] The simile recognition creation module creates multi-level simile recognition (MLSR) tasks to help PLMs evaluate simile quality; the MLSR score serves as a self-validation mechanism to align the simile knowledge used in different types of simile tasks.

[0121] The simile generation module integrates multi-level simile recognition scores into the simile generation task.

[0122] Furthermore, the specific working process of the dataset construction module is as follows: constructing metaphor data with multiple metaphor levels; and using the metaphor data to train PLMs to assign quality scores to each level.

[0123] Furthermore, in Figure 1 In the text, four levels of metaphor with different qualities are defined;

[0124] The first level is a correct simile, in which there are three healthy simile relationships between the components of the simile (as shown in the fourth column); the remaining levels are constructed by replacing the components of the simile and resulting in varying degrees of damage to the simile relationships.

[0125] The second level has two substitution methods. The first method involves replacing the metaphor with its superordinate word; for example, "The athlete runs as fast as a cheetah" is changed at this level to "The athlete runs as fast as an animal." The resulting sentence is still a simile, comparing two different things that share a property. However, it is less precise than the original sentence because not all animals "run fast." While understanding the sentence involves comparing "the athlete" and "an animal that can run fast," "a cheetah" is more specific than "an animal," making the simile more vivid and easier to understand. Figure 1 As shown in the fourth column, the relationship between shared attributes and metaphors may be altered and impaired as a result;

[0126] The second substitution method involves replacing the metaphor with a synonym of the subject; for example, "The athlete runs as fast as a cheetah." is changed to "The athlete runs as fast as a player." at this level. A simile requires comparing two things in different categories. "The athlete" and "a player" belong to the same category, thus breaking the simile relationship between the subject and the metaphor.

[0127] The third level replaces the metaphor or shared attribute according to the replacement rules; in Figure 1 In the sentence, "The athlete runs as fast as a cheetah," is replaced with "The athlete runs as fast as air" / "The athlete runs as green as a cheetah." When replacing the metaphor / shared attribute, the shared attribute "fast" still belongs to the subject "athlete," but is unrelated to the metaphor "air." When replacing the shared attribute, the sentence still compares two different things ("athlete" and "cheetah"), but the shared attribute "green" is incorrect. For example... Figure 1 As shown in the fourth column, in the third level, two of the three metaphorical relationships may be impaired.

[0128] The fourth level uses replacement rules to replace all shared attributes and metaphors. Figure 1 In the original text, "The athlete runs as fast as a cheetah" was changed to "The athlete runs as green as black eyes." Not only is "green" not a shared attribute between "athlete" and "black eyes," but "athlete" and "black eyes" also lack comparable attributes. At this level, all three metaphorical relationships are potentially compromised.

[0129] Furthermore, the replacement rules for the third and fourth layers are as follows: First, words with the same part of speech as the original words are used for replacement, that is, shared attributes are replaced with adjectives, while the subject / metaphor is replaced with nouns / phrases / sentences that can serve as verb subjects / objects;

[0130] Secondly, the relationships in ConceptNet (Robyn Speer, Joshua Chin, and Catherine Havasi. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017.) are used to select more appropriate words;

[0131] In the second level, the "IsA" relation in ConceptNet is used to find the hypernym of the subject / metaphor; in the fourth level, when replacing the metaphor, nouns / phrases that do not share attributes with the metaphor are selected; when replacing attributes, adjectives that are unrelated to the subject and metaphor are selected.

[0132] When a simile component contains more than one word, use the first noun or adjective in the component for replacement;

[0133] When no metaphor component is available in ConceptNet, or a suitable replacement cannot be found, only the first rule (replace with a word of the same part of speech as the original word) is followed, and a word is randomly selected from ConceptNet.

[0134] Furthermore, the working process of the metaphor recognition and creation module is as follows: this invention employs a multi-level contrastive learning method to train the PLM; for Figure 1 Each metaphor level in the model is assigned a reference score between 0 and 1 (1.000, 0.667, 0.333, 0.000) to reflect the quality of the metaphor. During training, the MLSR model learns to predict the score of the input sequence. The training objective is to minimize the distance between the predicted score and the reference score. It is important to note that a linear relationship is assumed between the reference quality score and the metaphor level. Figure 1 The relationship between metaphorical elements in the model is disrupted; this setting facilitates model training.

[0135] Figure 2 (a) shows the model structure of MLSR when using BERT as the backbone; the first output position (a special marker before the input sequence) is shown. <cls>The first decoded vector is used as the sequence representation to compute the multi-level SR loss. When the method is applied to other PLMs with decoder structures (Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020), the first decoded vector is used as the input representation to compute the MLSR loss. The input sequence is represented as , the set of input sequences is represented as D, and a sentence in D is represented as h. D h D A multilayer perceptron (MLP) is used to obtain the predicted quality score S of the input sequence D. D :

[0136] S D =σ(W2·μ(W1·h) D +b1)+b2),

[0137] Use the Sigmoid function and the tanh function respectively; Figure 1 Each level of metaphor is represented by D. i (i∈{1, 2, 3, 4}); then, the MLSR model is trained using separation loss and compactness loss.

[0138] Furthermore, the separation loss distinguishes different metaphor levels by different quality scores; for each metaphor level i, a center score is first calculated. in This is the quality score at this level, K. i D in the training batch i The number of examples; the separation loss between different metaphor levels is:

[0139]

[0140] Where λ = 0.333 is the lower bound of the distance between the two center scores, and ω = lj is the weight used to amplify the lower bound based on the metaphor quality gap;

[0141] The compactness loss compresses examples within the same metaphor level together, which acts as a normalization mechanism to avoid outliers at each level.

[0142] Specifically, for k∈{1, 2, ..., K} i For each example in}, its metaphorical quality score Forced to be associated with the corresponding center score Closer, as shown below:

[0143]

[0144] Where γ is the upper bound of the distance between the center point of a replacement level and the scores within that level; γ allows the model to automatically adjust the quality score of each level within a certain range; the final loss of MLSR is:

[0145]

[0146] Where N is the number of training samples; after training, the MLSR model can assign a metaphor quality score to the input sequence.

[0147] Furthermore, the specific working process of the metaphor generation module is as follows: Figure 2 As shown in (b), when using BERT as the backbone network, the SI and SG tasks are the same as the MLM tasks in BERT; where BERT learns to recover the masked attributes or metaphors; when using other PLMs with decoder structures (e.g., the T5 model) for the SI / SG tasks, the PLMs are provided with a task instruction and the masked sequence, and are required to generate the missing components; the loss function is:

[0148]

[0149] in, The t-th word representing the hidden attribute / metaphor (out of a total of r words, r≥1).

[0150] Train a single PLM to handle all three metaphor tasks: MLSR, SI, and SG; the loss function is... In SI / SG testing, following previous work, PLMs select missing attributes or metaphors from the candidate pool;

[0151] like Figure 2 As shown in (c), when experimenting with BERT, the masked word prediction head of BERT is used to calculate the probability of each candidate; the candidate with the highest probability will be selected as the final choice; when using other large PLMs with decoder structures for SI / SG tasks, an instruction is provided and the PLMs are asked to generate the missing components; the probability of each candidate is the generation probability.

[0152] To utilize MLSR results for self-validation, each candidate is concatenated to the input sequence (to the masking location) as a new input; the model can then compute the MLSR score for that candidate; the final score S for each candidate is... final It is the MLSR score S mlsr and SI / SG score S si / sg Weighted sum:

[0153] S final =β*S mlsr +(1-β)*S si / sg

[0154] Where β is a hyperparameter.

[0155] This invention significantly improves the accuracy of exploring metaphorical knowledge in pre-trained language models. Experiments on multiple metaphor datasets demonstrate that the system of this invention implements a self-verifying method for exploring metaphorical knowledge in pre-trained language models, achieving state-of-the-art performance with an accuracy improvement of over 5% compared to existing methods. This invention also tested methods with different PLMs (e.g., the T5 model in Example 1), showing that the self-verifying method is applicable to different types of PLMs and can be applied to language models with larger scales.

[0156] Example 3

[0157] This invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and processor are connected via a bus. Specifically, the processor implements any of the steps in Embodiment 1 by running the computer program stored in the memory.

[0158] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0159] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0160] As can be seen from the above, the electronic device provided in this embodiment of the invention can implement a self-verifying method for exploring simile knowledge in a pre-trained language model, as described in Embodiment 1, by running a computer program. This method integrates different types of simile knowledge in PLMs and makes better use of them. The invention also tested the method with different PLMs, and the results show that the self-verifying method is applicable to different types of PLMs and can be applied to language models with a larger scale.

[0161] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0162] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0163] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0164] It should be noted that the methods and detailed examples provided in the above embodiments can be incorporated into the apparatus and devices provided in the embodiments, and can be referred to each other, without further elaboration.

[0165] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0166] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0167] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.< / cls> < / cls>

Claims

1. A self-verifying method for exploring metaphorical knowledge in a pre-trained language model, characterized in that, The method includes the following steps: Step 1: Construct a multi-level metaphor recognition dataset; specifically, construct metaphor data with multiple metaphor levels; use the metaphor data to train PLMs and assign quality scores to each level. Step 2: Create a multi-level metaphor recognition task to help PLMs evaluate metaphor quality. Specifically, a multi-level contrastive learning method is used to train the PLM. For each metaphor level, a reference score between 0 and 1 is assigned. During training, the MLSR model learns to predict the score of the input sequence. The training objective is to minimize the distance between the predicted score and the reference score. Here, it is assumed that there is a linear relationship between the reference quality score and the metaphor level, which facilitates the model to automatically learn the quality scores at different levels. The model structure of MLSR when using BERT as the backbone; the first output position is used as the sequence representation; when applying the method to other PLMs with decoder structures, the first decoded vector is used as the input representation to compute the MLSR loss; the Sigmoid function and the tanh function are used respectively; each metaphor level is represented as D. i Then, the MLSR model is trained using separation loss and compactness loss. The separation loss distinguishes different metaphor levels by different quality scores representing different metaphor levels; for each metaphor level i, a center score is first calculated. Then calculate the separation loss between different levels of metaphor; Compactness loss compresses examples within the same metaphorical hierarchy together to avoid outliers at each level; Step 3: Integrate the multi-level metaphor recognition scores into the metaphor generation task; When using BERT as the backbone network, the SI and SG tasks are the same as the MLM tasks in BERT; where BERT learns to recover the masked attributes or metaphors; when using other PLMs with decoder structures for the SI / SG tasks, the PLMs are provided with a task instruction and a masked sequence, and are required to generate the missing components. Train a single PLM to handle all three metaphor tasks: MLSR, SI, and SG; the loss function is... In SI / SG testing, following previous work, PLMs select missing attributes or metaphors from the candidate pool; When experimenting with BERT, the masked word prediction head of BERT is used to calculate the probability of each candidate; the candidate with the highest probability is selected as the final choice; when using other large PLMs with decoder structures for SI / SG tasks, an instruction is provided and the PLMs are asked to generate the missing components; the probability of each candidate is the generation probability. To use the MLSR results for self-validation, each candidate is concatenated to the input sequence as a new input; the model can then compute the MLSR score for that candidate.

2. The method according to claim 1, characterized in that, Define four levels of metaphor with different qualities; The first level is a correct simile, in which there are three healthy simile relationships between the components of the simile; the remaining levels are constructed by replacing the components of the simile, resulting in varying degrees of damage to the simile relationships. The second level has two replacement methods. The first replacement method is to replace the metaphor with its superordinate word. The second substitution method is to replace the metaphor with a synonym of the subject; The third level replaces the metaphor or shared attribute according to the replacement rules; The fourth level uses replacement rules to replace all shared attributes and metaphors.

3. The method according to claim 2, characterized in that, The replacement rule firstly replaces words with words of the same part of speech as the original words, that is, shared attributes are replaced with adjectives, while the subject / metaphor is replaced with nouns / phrases / sentences that can serve as verb subjects / objects; Secondly, we can use the relationships in ConceptNet to select more suitable words; In the second level, the "IsA" relation in ConceptNet is used to find the hypernym of the subject / metaphor; in the fourth level, when replacing the metaphor, nouns / phrases that do not share attributes with the metaphor are selected; when replacing attributes, adjectives that are unrelated to the subject and metaphor are selected. When a simile component contains more than one word, use the first noun or adjective in the component for replacement; When no metaphor component is available in ConceptNet, or a suitable replacement cannot be found, only the first rule is followed, and words are randomly selected from ConceptNet.

4. The method according to claim 2, characterized in that, Step 2 specifically involves the reference scores between 0 and 1 being 1.000, 0.667, 0.333, and 0.

000. Let the input sequence be denoted as , let the set of input sequences be denoted as D, and let a sentence in D be denoted as h. D h D A multilayer perceptron is used to obtain the predicted quality score S of the input sequence D. D The formula is: S D =σ(W2·μ(W1·h D +b1)+b2), Each metaphor level is specifically represented as D. i (i∈{1, 2, 3, 4}).

5. The method according to claim 4, characterized in that, The separation loss is as follows: For each metaphor level i, first calculate a center score. in, This is the quality score for this level, K. i D in the training batch i The number of examples; the separation loss between different metaphor levels is: Where λ = 0.333 is the lower bound of the distance between the two center scores, and ω = lj is the weight used to amplify the lower bound based on the metaphor quality gap; The compactness loss is: for k∈{1,2,...,K} i For each example in}, its metaphorical quality score Forced to be associated with the corresponding center score Closer, as shown below: Where γ is the upper bound of the distance between the center point of a replacement level and the scores within that level; γ allows the model to automatically adjust the quality score for each level; the final loss of MLSR is: Where N is the number of training samples; after training, the MLSR model can assign a metaphor quality score to the input sequence.

6. The method according to claim 1, characterized in that, Step 3 specifically involves the loss function being: in, The t-th word representing the hidden attribute / metaphor; Based on the candidate's MLSR score, calculate the final score S for each candidate. final It is the MLSR score S mlsr and SI / SG score S si / sg Weighted sum: S final =b * S mlsr +(1-b) * S si / sg Where β is a hyperparameter.

7. A self-verifying system for exploring metaphorical knowledge in a pre-trained language model, characterized in that, The system uses a self-verifying method for exploring metaphorical knowledge in a pre-trained language model as described in any one of claims 1-6, and the system comprises, The dataset construction module builds a multi-level metaphor recognition dataset. The simile recognition creation module allows for the creation of multi-level simile recognition tasks, helping PLMs evaluate simile quality. MLSR scores serve as a self-validation mechanism to align metaphorical knowledge used in different types of metaphorical tasks; The simile generation module integrates multi-level simile recognition scores into the simile generation task.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images