A method for exploring the humor comprehension ability of pre-trained language models

Abstract

This invention provides a method for exploring the humor comprehension ability of a pre-trained language model, which can systematically guide humor generation tasks. The method employs a human-machine combined humor comprehension evaluation mechanism to assess the humor comprehension ability of the pre-trained language model across four dimensions: humor recognition, humor type classification, humor level classification, and joke detection. In this method, the humor comprehension evaluation mechanism not only probes the humor comprehension ability of the original pre-trained language model, thus evaluating whether the original pre-trained language model can understand humor before or after fine-tuning, but also probes the humor comprehension ability of the pre-trained language model under knowledge enhancement, thereby assessing whether external knowledge helps improve the humor comprehension ability of the pre-trained language model. Simultaneously, the evaluation results of the humor comprehension ability of the pre-trained language model are interpreted, probing the language model's humor comprehension ability from three perspectives: whether the pre-trained language model can detect interpretable cue words to understand humor.

Description

Technical Field

[0001] This invention belongs to the field of machine higher-order cognition, specifically involving a method for exploring the humor comprehension ability of pre-trained language models for machine higher-order cognition. Background Technology

[0002] Imbuing machines with a human sense of humor is an extremely challenging task. With the development of human-computer interaction systems and applications, the ability to enable machines to possess a sense of humor may foreshadow the completion of the Tower of Babel in human-computer interaction. Existing humor-related tasks primarily focus on humor recognition and humor generation. Early work utilized surface language features and templates to imitate or generate humor. For example, humor was identified through word association and underlying grammatical structures, or puns were generated by analyzing language structure and using retrieval and editing methods. However, these methods heavily rely on the prohibitively expensive human resources required to design templates for diverse datasets and can only recognize or generate a very limited range of humorous expressions.

[0003] With the rise of pre-trained language models, some works have made initial attempts to utilize them for humor recognition and generation. Due to their powerful understanding and generation capabilities, pre-trained language models can recognize or generate more types of humorous expressions. However, these works have not explored whether pre-trained language models can understand humor. Humor understanding is fundamental to humor recognition and generation; only when pre-trained language models have a more comprehensive understanding of humor are they more likely to improve their humor recognition and generation capabilities.

[0004] Early work primarily utilized surface-level linguistic features and templates to mimic or generate humor. However, these methods relied on prohibitively expensive human resources to design templates for diverse datasets and could only recognize or generate a very limited range of humorous expressions. While some works attempted to use pre-trained language models for humor recognition and generation, they did not explore whether these models could understand humor. Humor understanding is fundamental to both humor recognition and generation; pre-trained language models are more likely to improve their humor recognition and generation capabilities only when they have a more comprehensive understanding of humor. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method for exploring the humor comprehension capabilities of pre-trained language models for higher-order machine cognition. The technical solution adopted in this invention is as follows:

[0006] This invention provides a method for exploring the humor comprehension ability of a pre-trained language model, characterized by the following steps: Step 1, constructing a humor comprehension evaluation mechanism, which has four tasks: humor recognition, humor type classification, humor level classification, and joke detection; Step 2, probing the humor comprehension ability of the original pre-trained language model to be tested based on the humor comprehension evaluation mechanism; Step 3, injecting different types of external knowledge into the original pre-trained language model to be tested to obtain a knowledge-enhanced pre-trained language model to be tested, and probing the humor comprehension ability of the knowledge-enhanced pre-trained language model to be tested based on the humor comprehension evaluation mechanism; Step 4, interpreting the humor comprehension ability of the pre-trained language model, that is, exploring whether the original pre-trained language model to be tested and the knowledge-enhanced pre-trained language model to be tested can detect cues for understanding humor in humorous texts.

[0007] This invention provides a method for exploring the humor comprehension ability of a pre-trained language model, which may also have the following technical features: a humor recognition task to distinguish humorous text from non-humorous text; a humor type classification task to distinguish humor types, including at least pun-based humor, ambiguous humor, and absurd humor; a humor level classification task to distinguish humor levels, which are divided into weak humor, moderate humor, and strong humor based on the number of humorous elements; and a joke detection task to identify whether there is semantic inconsistency between the context and the humorous ending, which is evaluated manually from three aspects: semantics, correctness, and readability, and evaluated by machine using a contrastive learning method from the similarity between non-humorous and humorous endings. Texts with low scores in both manual and machine evaluations are considered to have insufficient jokes and need to be regenerated using a generative model.

[0008] The present invention provides a method for exploring the humor comprehension ability of a pre-trained language model, which may also have the following technical features, wherein step 2 includes the following steps: step 2-1, selecting several representative original pre-trained language models; step 2-2, taking the text to be tested as input, and using the original pre-trained language models to perform humor recognition and detection to output the attributes of the text to be tested, which include humor or non-humor, humor type, and humor registration; step 2-3, taking the preceding and ending sentences of the text to be tested as sentence pairs and inputting them into the original pre-trained language models, calculating the similarity score of the sentence pairs, and thereby determining whether the ending of the text to be tested is a humorous ending.

[0009] The present invention provides a method for exploring the humor understanding ability of a pre-trained language model, which may also have the following technical features: the similarity score of sentence pairs is calculated based on sentence embedding and an online contrastive loss function is used. When the similarity score of a sentence pair exceeds a predetermined threshold, the ending is regarded as a non-humorous ending; otherwise, it is regarded as a humorous ending.

[0010] The present invention provides a method for exploring the humor understanding ability of a pre-trained language model, which may also have the following technical features, wherein the specific process of step 3 is as follows: Step 3-1, obtain the character embedding of each Chinese character in the input text based on knowledge and common sense based on the existing knowledge base and common sense base respectively; Step 3-2, obtain the pinyin embedding of each Chinese character in the input text based on linguistic knowledge to detect different semantics of Chinese characters with the same or similar pronunciations; Step 3-3, use a fusion layer to enhance the sentence embedding representation of the pre-trained language model, and use a fully connected layer to normalize the pinyin embedding along the letter dimension of each pinyin; Step 3-4, add the normalized pinyin embedding to the sentence embedding of the pre-trained language model, and then output the prediction result through a classifier.

[0011] The present invention provides a method for exploring the humor understanding ability of a pre-trained language model, which may also have the following technical features, wherein step 3-2, obtaining the pinyin embedding, is as follows: First, a pinyin package tool is used to generate pinyin containing tones for each Chinese character in the given text. For polyphonic characters, the first pronunciation output by the pinyin package tool is selected. The maximum length of the input pinyin sequence is set to 8, and the special character "-" is used to pad pinyin sequences with a length less than 8. A convolutional neural network is used to obtain the pinyin embedding of the input text. This process is represented as follows:

[0012] Emb pinyin =Maxpool(CNN(pypinyin(Seq)) in )))

[0013] In the formula, Seq in The input text sequence is represented by pypinyin, which means converting the text sequence into a pinyin sequence. CNN means extracting features of the pinyin sequence through the convolutional layers of a convolutional neural network. Maxpool means compressing the features through a max pooling layer to remove redundant information to obtain the final pinyin embedding.

[0014] The present invention provides a method for exploring the humor understanding ability of a pre-trained language model, which may also have the following technical features: step 4 uses the Captum package to explain the humor understanding ability of the pre-trained language model from two aspects: gradient-based and perturbation-based.

[0015] The present invention provides a method for exploring the humor understanding ability of a pre-trained language model, which may also have the following technical features: interpreting the humor understanding ability of the pre-trained language model based on gradient means: calculating the saliency map through the gradient of the input relative to the output, the process is as follows: first, the word embedding of each character is used as the input of the pre-trained language model, then L2 regularization is used to normalize the word embedding, then Input X Gradient is used to multiply the gradient with the normalized word embedding to improve the discriminativeness of the saliency score, and finally a visualization tool is used to draw the saliency map of the saliency score.

[0016] This invention provides a method for exploring the humor comprehension ability of a pre-trained language model, which also has the following technical features: interpreting the humor comprehension ability of a pre-trained language model based on perturbation involves perturbing the input to discover which input regions have a significant impact on prediction. The process is as follows: First, the credibility of the saliency map is investigated to detect whether the correct predictions made by the pre-trained language model are randomly selected. Then, the stability of the saliency map is tested to detect whether changes in the prediction results affect the saliency map. That is, Chinese characters that do not change the original text are added to the end of each input text. The correlation between changes in prediction accuracy and changes in saliency score is tested using Pearson and Spearman correlation coefficients. If the p-value in the correlation test is less than 0.05, it proves that there is a statistically significant difference between changes in prediction accuracy and changes in saliency score, meaning that the saliency score highly follows the prediction results. Therefore, the saliency score is stable, and the cue words used to explain the humor comprehension ability of the pre-trained language model can be trusted by humans.

[0017] Invention Function and Effect

[0018] According to a method for exploring the humor comprehension ability of a pre-trained language model, this invention employs a human-computer combined humor comprehension evaluation mechanism to assess the pre-trained language model across four dimensions: humor recognition, humor type classification, humor level classification, and joke detection. In this method, the humor comprehension evaluation mechanism not only probes the humor comprehension ability of the original pre-trained language model, thus evaluating whether the original pre-trained language model can understand humor before or after fine-tuning, but also probes the humor comprehension ability of the pre-trained language model under knowledge enhancement, thereby assessing whether external knowledge helps improve the humor comprehension ability of the pre-trained language model. Simultaneously, the evaluation results of the humor comprehension ability of the pre-trained language model are interpreted, probing the language model's humor comprehension ability from three perspectives: whether the pre-trained language model can detect interpretable cue words to understand humor.

[0019] The present invention explores the humor comprehension ability of pre-trained language models, focusing on the evaluation of text humor and the detection of the model's ability to understand humor. Compared with the limitations of existing technologies that mainly focus on how to produce high-quality humorous texts, the present invention can more systematically guide the humor generation task.

[0020] Furthermore, existing machine humor generation mainly relies on templates. Although it can produce some high-quality content, it requires a lot of human intervention and lacks creativity. In contrast, this invention can help generate new expressions by fine-tuning a pre-trained language model and subjecting it to quality assessment constraints, thereby generating innovative and high-quality humorous text. Attached Figure Description

[0021] Figure 1 This is a framework diagram of the method for exploring the humor comprehension ability of a pre-trained language model in this embodiment of the invention. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following describes in detail a method for exploring the humor comprehension ability of a pre-trained language model, in conjunction with embodiments and accompanying drawings.

[0023] <Example>

[0024] Figure 1 This is a framework diagram of the method for exploring the humor comprehension ability of a pre-trained language model in this embodiment of the invention.

[0025] like Figure 1 As shown in the figure, this embodiment proposes a humor comprehension evaluation mechanism to explore the humor comprehension ability of pre-trained language models. The humor comprehension evaluation mechanism has four tasks: humor recognition, humor type classification, humor level classification, and joke detection.

[0026] The humor recognition task aims to distinguish between humorous and non-humorous text. For the task dataset, humorous and non-humorous texts of 20-200 characters in length with similar semantic styles can be selected.

[0027] The humor type classification task aims to distinguish humor types, including puns, ambiguity, and paradox. Specifically, puns are humor where certain words or phrases in the humorous text have the same or similar pronunciation but different meanings; ambiguity is humor where certain words or phrases in the humorous text have multiple interpretations, which are used simultaneously in the same text; and paradox is humor where there is semantic inconsistency in the humorous text.

[0028] The humor classification task aims to distinguish humor levels, including weak humor, moderate humor, and strong humor. Among them, weak humor is closer to dry humor and contains the fewest humorous elements, while strong humor contains the most humorous elements.

[0029] The joke detection task is used to identify semantic inconsistencies between the preceding text and the humorous ending. Specifically, it involves human evaluation based on semantics, correctness, and readability, and machine evaluation based on the similarity between the non-humorous and humorous endings using contrastive learning. Texts scoring low in either human or machine evaluation are considered to lack sufficient humor and require regeneration using a generative model.

[0030] This embodiment studies the humor comprehension ability of pre-trained language models from the following three aspects: (1) whether the pre-trained language model can understand humor before or after fine-tuning; (2) whether external knowledge helps to improve the humor comprehension ability of pre-trained language models; and (3) whether the pre-trained language model can detect interpretable cue words to understand humor.

[0031] To answer the above questions, this embodiment proposes a three-step evaluation method and uses the aforementioned four humor-related tasks to systematically evaluate the pre-trained language model. Furthermore, a Chinese humor dataset has been constructed, which fully meets the requirements of the four tasks and the three-step evaluation method.

[0032] In this embodiment, the three-step evaluation method for probing the humor comprehension ability of a pre-trained language model is as follows:

[0033] Step 1: Detecting the humor comprehension ability of the original pre-trained language model.

[0034] like Figure 1 As shown in (a), this study examines whether the original pre-trained language model possesses humor comprehension capabilities, and whether fine-tuning the pre-trained language model on a humor dataset enhances its humor comprehension ability. Specifically:

[0035] Several representative pre-trained language models were selected, and their humor comprehension abilities were explored through the above four humor-related tasks.

[0036] Since the first three tasks belong to text classification, a piece of text is used as input to the original pre-trained language model, and the model outputs the attributes of the text, including whether it is humorous or not, the type of humor, and the level of humor.

[0037] The final task involves text matching, where the preceding and concluding sentences are combined into sentence pairs and input into a pre-trained language model. The model then outputs the similarity score of these sentence pairs to determine whether the ending is humorous. In this embodiment, the similarity score of sentence pairs is calculated based on sentence embeddings, using an online contrastive loss function. If the similarity score of a sentence pair exceeds a set threshold, the ending is considered non-humorous. Otherwise, it is considered humorous.

[0038] Step 2: Detecting the humor comprehension ability of a knowledge-enhanced pre-trained language model.

[0039] like Figure 1 As shown in (b), different types of external knowledge are injected into the original pre-trained language model to obtain a knowledge-enhanced pre-trained language model. This allows us to explore which types of knowledge can help improve the humor comprehension ability of the pre-trained language model. Specifically:

[0040] First, based on the existing knowledge base and common sense base, the character embedding of each Chinese character in the input text is obtained based on knowledge and common sense.

[0041] Secondly, based on linguistic knowledge, such as Pinyin, the Pinyin embedding of each Chinese character in the input text is obtained to detect different semantics of Chinese characters with the same or similar pronunciations. In this embodiment, the PyPinyin package is used to generate Pinyin including tones for each Chinese character in the given text. For polyphonic characters, the first pronunciation output by the PyPinyin package is selected. The maximum length of the input Pinyin sequence is set to 8, and Pinyin sequences shorter than 8 are padded with the special character "-".

[0042] Then, a convolutional neural network is used to obtain the pinyin embedding of the input text. This is represented as follows:

[0043] Emb pinyin =Maxpool(CNN(pypinyin(Seq)) in )))

[0044] In the formula, Seq in The input text sequence is represented by pypinyin, which means converting the text sequence into a pinyin sequence. CNN means extracting features of the pinyin sequence through the convolutional layers of a convolutional neural network. Maxpool means compressing the features through a max pooling layer to remove redundant information to obtain the final pinyin embedding.

[0045] Next, a fusion layer is designed to enhance the sentence embedding representation of the pre-trained language model, and a fully connected layer is used to normalize the pinyin embedding along the letter dimension of each pinyin (since each pinyin is a sequence of letters).

[0046] Finally, the normalized pinyin embedding is added to the sentence embedding of the pre-trained language model, and then the result is output through a classifier.

[0047] Step 3: Explain the humor comprehension ability of the pre-trained language model.

[0048] To explore whether the humor comprehension ability of pre-trained language models conforms to human perception, this embodiment investigates whether pre-trained language models (including original pre-trained language models and knowledge-enhanced pre-trained language models) can detect appropriate cue words in humorous texts to understand humor.

[0049] like Figure 1 As shown in (c), this embodiment uses the Captum package to explain the humor understanding ability of the pre-trained language model from two aspects: gradient-based and perturbation-based.

[0050] Gradient-based methods refer to calculating the saliency map by using the gradient of the input relative to the output.

[0051] The specific process is as follows:

[0052] First, the differentiable word embeddings of each character are used as input to the pre-trained language model. Second, L2 regularization is applied to normalize the word embeddings. Then, Input X Gradient is used to multiply the gradient with the normalized word embeddings to improve the discriminative power of the saliency scores. Finally, a saliency plot of the saliency scores is drawn using a visualization tool.

[0053] Perturbation-based methods involve perturbing the input to identify which regions of the input have a significant impact on the prediction. The process is as follows:

[0054] First, the credibility of the saliency map is investigated to detect whether the correct predictions made by the pre-trained language model are based on random selection. In this embodiment, the top N (N=3) most salient Chinese characters are replaced with mask labels, and then the performance degradation of the pre-trained language model is measured. The more significant the degradation, the higher the credibility of the saliency map.

[0055] Next, the stability of the saliency plot is tested to determine whether changes in the prediction results affect the saliency plot. In this embodiment, some unimportant Chinese characters (i.e., characters that do not change the semantics of the input text) are added to the end of each input text. Then, the correlation between changes in prediction accuracy and changes in saliency score is detected based on the Pearson correlation coefficient and the Spearman correlation coefficient.

[0056] If the p-value in the correlation test is less than 0.05, it proves that there is a statistically significant difference between the change in prediction accuracy and the change in significance score. That is, the significance score highly follows the prediction result, so the significance score is stable. The cue words used to explain the humor ability of the pre-trained language model can be trusted by humans.

[0057] Functions and effects of the embodiments

[0058] This embodiment provides a method for exploring the humor comprehension ability of a pre-trained language model. The method employs a human-computer combined humor comprehension evaluation mechanism to assess the pre-trained language model across four dimensions: humor recognition, humor type classification, humor level classification, and joke detection. In this method, the humor comprehension evaluation mechanism not only probes the humor comprehension ability of the original pre-trained language model, thus evaluating whether the original pre-trained language model can understand humor before or after fine-tuning, but also probes the humor comprehension ability of the pre-trained language model under knowledge enhancement, thereby assessing whether external knowledge helps improve the humor comprehension ability of the pre-trained language model. Furthermore, the evaluation results of the humor comprehension ability of the pre-trained language model are interpreted, probing the language model's humor comprehension ability from three perspectives: whether the pre-trained language model can detect interpretable cue words to understand humor.

[0059] In this embodiment, the evaluation and detection model of textual humor is mainly based on its ability to understand humor. Compared with the limitations of existing technologies that mainly focus on how to produce high-quality humorous text, the method of this embodiment can more systematically guide the humor generation task.

[0060] Furthermore, existing machine humor generation mainly relies on templates. Although it can produce some high-quality content, it requires a lot of human intervention and lacks creativity. This embodiment, on the other hand, can help generate new expressions by fine-tuning a pre-trained language model and subjecting it to quality assessment constraints, thereby generating innovative and high-quality humorous text.

[0061] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

Claims

1. A method for exploring humor understanding capability of a pre-trained language model, characterized in that, The method comprises the following steps: Step 1, constructing a humor understanding evaluation mechanism with four tasks of humor recognition, humor type classification, humor level classification and punchline detection; Step 2, detecting the humor understanding ability of the original pre-trained language model to be tested based on the humor understanding evaluation mechanism; Step 3, injecting different types of external knowledge into the original pre-trained language model to be tested to obtain a knowledge-enhanced pre-trained language model to be tested, and detecting the humor understanding ability of the knowledge-enhanced pre-trained prediction model based on the humor understanding evaluation mechanism; Step 4, explaining the humor understanding ability of the pre-trained language model, that is, exploring whether the original pre-trained language model to be tested and the knowledge-enhanced pre-trained language model to be tested can detect clues for understanding humor in humorous texts, Wherein, the step 2 comprises the following steps: Step 2-1, selecting a plurality of representative original pre-trained language models; Step 2-2, inputting the test text as input, and detecting the humor recognition of the original pre-trained language model to output the attributes of the test text, including humor or non-humor, humor type and humor level; Step 2-3, inputting the beginning and ending sentences in the test text into the original pre-trained language model to calculate the similarity score of the sentence pair, so as to determine whether the ending of the test text is a humorous ending, Wherein, the specific process of step 3 is as follows: Step 3-1, based on the knowledge base and common sense base, respectively obtaining the knowledge and common sense based character embedding of each Chinese character in the input text; Step 3-2, based on linguistic knowledge, obtaining the pinyin embedding of each Chinese character in the input text to detect different semantics of Chinese characters with the same or similar pronunciation; Step 3-3, using a fusion layer to enhance the sentence embedding representation of the pre-trained language model, and using a fully connected layer to normalize the pinyin embedding along the letter dimension of each pinyin; Step 3-4, adding the normalized pinyin embedding and the sentence embedding of the pre-trained language model, and then outputting the prediction result through the classifier, Wherein, the process of obtaining pinyin embedding in step 3-2 is as follows: First, use the pinyin tool to generate pinyin with tone for each Chinese character in the given text, for multi-sound characters, select the first pronunciation output by the pinyin tool, set the maximum length of the input pinyin sequence to 8, use the special character "-" to pad the pinyin sequence with a length less than 8, and use a convolutional neural network to obtain the pinyin embedding of the input text, which is represented as follows: ， In the formula, Seq in represents the input text sequence, pypinyin represents the conversion of the text sequence into a pinyin sequence, CNN represents the extraction of features of the pinyin sequence through the convolution layer of the convolutional neural network, Maxpool represents the compression of the features through the maximum pooling layer to remove redundant information to obtain the final pinyin embedding, Wherein, step 4 uses Captum package to explain the humor understanding ability of the pre-trained language model from the aspects of gradient and perturbation.

2. The method of exploring the humor understanding ability of the pre-trained language model according to claim 1, wherein: wherein, The humor recognition task is used to distinguish humorous texts and non-humorous texts; The humor type classification task is used to distinguish the types of humor, including at least homophonic humor, ambiguous humor and reversal humor; The humor level classification task is used to distinguish the level of humor, which is divided into weak humor, medium humor and strong humor according to the number of laughing elements; The punchline detection task is used to identify whether there is semantic inconsistency between the context and the end of the humor, which is manually evaluated from the aspects of semanticity, correctness and readability, and machine evaluated from the similarity of non-humor endings and humor endings using contrastive learning. For texts with less scores in manual or machine evaluation, it is considered that the punchline is insufficient, and a new expression needs to be generated using a generation model.

3. The method of claim 1, wherein: wherein, The similarity score of the sentence pair is calculated based on sentence embedding, and an online contrastive loss function is used. When the similarity score of the sentence pair exceeds a predetermined threshold, the ending is considered as a non-humor ending, otherwise it is considered as a humor ending.

4. The method of claim 1, wherein: wherein The explanation of the humor understanding ability of the pre-trained language model based on gradient refers to the calculation of the saliency map through the gradient of the input relative to the output, and the process is as follows: First, the word embedding of each word is used as the input of the pre-trained language model, then the word embedding is normalized using L2 regularization, then the gradient is multiplied with the normalized word embedding using Input X Gradient to improve the discrimination of the saliency score, and then the saliency map of the saliency score is drawn using a visualization tool.

5. The method of claim 4, wherein: wherein, The explanation of the humor understanding ability of the pre-trained language model based on perturbation refers to the perturbation of the input to find out which input area has a significant impact on the prediction, and the process is as follows: First, investigate the reliability of the saliency map to detect whether the correct prediction made by the pre-trained language model is randomly selected, then detect the stability of the saliency map to detect whether the change of the prediction result will affect the saliency map, that is, add Chinese characters that will not change the original language at the end of each input text, and detect the correlation between the change of the prediction accuracy and the change of the saliency score according to the Pearson correlation coefficient and the Spearman correlation coefficient, If the p value in the correlation detection is less than 0.05, it proves that there is a statistical difference between the change of the prediction accuracy and the change of the saliency score, that is, the saliency score highly conforms to the prediction result, so the saliency score has stability, and the clue word that explains the humor ability of the pre-trained language model can be trusted by humans.

Images