An emoji classification method with metaphor information based on contrastive learning

Abstract

The application provides an emoji classification method with metaphor information based on contrast learning, which comprises the following steps: extracting effective data from a data set and performing preprocessing to obtain an emoji data set, dividing the emoji data set into a training set, a verification set and a test set; obtaining emoji text feature representation, emoji picture feature representation and emoji label information feature representation; calculating a picture-text joint representation feature vector and a label feature vector; constructing an emoji sentiment analysis training model based on contrast learning with metaphor information; training the emoji sentiment analysis training model by using the training set; verifying and testing the trained emoji sentiment analysis training model by using the verification set and the test set to obtain an emoji sentiment analysis model; and inputting an emoji to be analyzed into the emoji sentiment analysis model to obtain a sentiment classification result and an intention detection classification result of the emoji to be analyzed. The application can improve the accuracy of emoji sentiment classification and intention detection classification.

Description

Technical Field

[0001] This invention belongs to the field of information analysis, and specifically discloses a method for classifying emojis with metaphorical information based on contrastive learning. Background Technology

[0002] The spread and popularity of emojis have grown rapidly in recent years. People's way of expressing emotions online has gradually shifted from text to emojis—a multimodal form combining text and images. Therefore, capturing the true semantics from these emojis is crucial, as it aids many natural language processing tasks such as question answering, sentiment analysis, and intent detection. Traditional methods for multimodal feature fusion involve concatenation, which directly concatenates the feature vectors obtained from the text and image of the emoji. This approach doesn't consider the rich emotional information within emojis and loses some important features during the concatenation process. Furthermore, the emotional information described by this method has little correlation with the emoji itself, resulting in inaccurate emoji classification results.

[0003] This invention selects the DistilBERT encoder for text encoding. This encoder is a model based on BERT with knowledge distillation. Compared to BERT, DistilBERT has fewer parameters and faster computation. After inputting word vectors from a sentence into the encoder, the vector corresponding to [CLS] is generally selected for the output representation of the classification task, resulting in a 768-dimensional vector. For image encoding, this invention selects the ResNet-50 encoder. ResNet-50 has achieved excellent results in ImageNet image classification, so it also performs well in encoding emoji images. After the model's adaptive average pooling layer, the final representation is a 2048-dimensional vector. In this invention, a projection layer method is used to further extract feature vectors. The projection layer method was first proposed in SimCLR. A projection layer is added between the feature vector calculated by the encoder and the vector to be compared to further extract features, avoiding the loss of some important features during the comparison process. After the features extracted by the projection layer, feature fusion is performed on these features for further learning.

[0004] Learning the correlation between text and images through contrastive learning is a feasible method. In this invention, the correlation between the emotional information describing an emoji and the emoji itself can also be achieved through contrastive learning. The main idea of ​​contrastive learning is to maximize the relevant pairings and minimize the rest. In this method, the information describing the emotion of an emoji is considered a positive sample with the corresponding emoji information, while the rest are considered negative samples, thereby improving the accuracy of emoji sentiment analysis. Summary of the Invention

[0005] This invention proposes a contrastive learning-based method for classifying emojis with metaphorical information to address the problem of inaccurate emoji classification results in existing emoji analysis.

[0006] This invention provides a method for classifying emojis with metaphorical information based on contrastive learning, comprising the following steps:

[0007] S1. Extract valid data from the dataset and preprocess it to obtain an emoji dataset that integrates text information, Chinese and English emoji images, and tag information. Divide the emoji dataset into a training set, a validation set, and a test set. The valid data includes Chinese and English emoji images, emoji text information, emoji metaphor information, emoji sentiment information, and emoji intent information.

[0008] S2. Input the integrated text information from the emoji dataset obtained in step S1 into the text encoder to obtain the emoji text feature representation; input the Chinese and English emoji images into the image encoder to obtain the emoji image feature representation; input the tag information into the text encoder to obtain the emoji tag information feature representation.

[0009] S3. Using the text feature representation, image feature representation, and tag information feature representation of the emoticon obtained in step S2, calculate the image-text joint representation feature vector and the tag feature vector;

[0010] S4. Based on the image-text joint representation feature vector and label feature vector obtained in step S3, construct an emoji sentiment analysis training model with metaphorical information based on contrastive learning;

[0011] S5. Use the training set obtained in step S1 to train the emoji sentiment analysis training model constructed in step S4, and save the trained emoji sentiment analysis training model.

[0012] S6. Use the validation set and test set obtained in step S1 to validate and test the emoji sentiment analysis training model trained in step S5, and obtain the emoji sentiment analysis model.

[0013] S7. Input the emoji to be analyzed into the emoji sentiment analysis model obtained in step S6 to obtain the sentiment classification result and intent detection classification result of the emoji to be analyzed.

[0014] According to some embodiments of this application, a method for classifying emojis with metaphorical information based on contrastive learning is provided. In step S1, the metaphorical information of the emojis includes source metaphorical information and target metaphorical information. The emotional information of the emojis includes happiness, friendliness, anger, sadness, fear, dislike, and surprise. The intentional information of the emojis includes interaction, expression, entertainment, offense, and others.

[0015] According to some embodiments of this application, a method for classifying emojis with metaphorical information based on contrastive learning is provided. In step S1, the preprocessing includes segmenting the emoji text information and emoji metaphorical information using segmentation markers [SEP] to obtain integrated text information; and converting the emoji emotional information and emoji intent information into tag information, wherein the tag information includes sentences describing the emoji emotional information and sentences describing the emoji intent information.

[0016] According to some embodiments of this application, a method for classifying emojis with metaphorical information based on contrastive learning, step S2 includes:

[0017] S201. Input the integrated text information into the tokenizer of the DistilBERT encoder to obtain the input number and attention flag of the corresponding text. The input number represents the result of mapping each word vector to a number after the integrated text information is segmented by the tokenizer. The attention flag uses the numbers 1 and 0 to represent different information. 1 represents information that needs attention in the input number, and 0 represents information that does not need attention in the input number.

[0018] S202. Input the input number and attention flag obtained in step S201 into the pre-trained DistilBERT encoder, select the output vector corresponding to the special symbol [CLS] inserted before the integrated text information to represent the semantic representation of the integrated text information, and obtain a 768-dimensional emoji text feature representation;

[0019] S203. Input the Chinese and English emoticon images into cv2.imread for reading, convert the reading result into RGB format and input it into the albumentations image data augmentation library for reconstruction and normalization to obtain a tensor with a (3,224,224) structure.

[0020] S204. Input the tensor with the (3,224,224) structure obtained in step S203 into the pre-trained ResNet50 image encoder to obtain a 2048-dimensional feature representation of the emoji image;

[0021] S205. Input the tag information into the word segmenter of the DistilBERT encoder, and repeat steps S201-S202 to obtain a 768-dimensional feature representation of the emoji tag information.

[0022] According to some embodiments of this application, a method for classifying emojis with metaphorical information based on contrastive learning, step S3 includes:

[0023] S301. The 768-dimensional emoji text feature representation obtained in step S202 is passed through a fully connected layer to convert the input dimension of 768 dimensions into an output dimension of 256 dimensions of emoji text vector. The emoji text vector is then passed through a GELU activation function and a fully connected layer with an output dimension of 256 dimensions. The result is then passed through a dropout_rate=0.20 blocking feature detector to obtain an emoji text blocking feature vector. The emoji text blocking feature vector is then added to the emoji text vector. The result of the addition operation is normalized by LayerNorm to obtain the text feature.

[0024] The 2048-dimensional emoji image feature representation obtained in step S204 is passed through a fully connected layer to transform the input dimension of 768 dimensions into an output dimension of 256 dimensions for the emoji image vector. The emoji image vector is then passed through a GELU activation function and a fully connected layer with an output dimension of 256 dimensions. The result is then passed through a dropout_rate=0.20 blocking feature detector to obtain the emoji image blocking feature vector. The emoji image blocking feature vector is then added to the emoji image vector. The result of the addition operation is normalized using LayerNorm to obtain the image features.

[0025] S302. Add the image features and text features obtained in step S301 to obtain the image-text joint representation feature vector it. f ;

[0026] S303. The 768-dimensional emoji tag information feature representation obtained in step S305 is passed through a fully connected layer to transform the input dimension of 768 dimensions into an output dimension of 256 dimensions, resulting in the tag feature vector l. f .

[0027] According to some embodiments of this application, a method for classifying emojis with metaphorical information based on contrastive learning, step S4 includes:

[0028] S401. Calculate the cosine similarity between the image-text joint representation feature vector and the label feature vector to obtain the cosine similarity matrix logits, as shown in formula (1):

[0029] logits=np.dot(it f ,l f ) np.exp(t) (1)

[0030] Where t is a learnable parameter;

[0031] The diagonal elements of the cosine similarity matrix represent the calculation results of the image-text joint representation feature vector and the corresponding tag feature vector, and the remaining elements of the cosine similarity matrix are the calculation results of the image-text joint representation feature vector and the non-corresponding tag feature vector.

[0032] S402. Construct a loss function based on the idea of ​​contrastive learning, as shown in formula (2):

[0033] (2)

[0034] Where T represents transpose. To calculate the cross-entropy loss, the label is calculated as shown in formula (3):

[0035] (3)

[0036] By maximizing the diagonal elements of the cosine similarity matrix and minimizing the remaining elements of the cosine similarity matrix using the loss function, a sentiment analysis model for emojis with metaphorical information based on contrastive learning is obtained.

[0037] According to some embodiments of this application, a method for classifying emojis with metaphorical information based on contrastive learning is provided. In step S7, the emoji to be analyzed is input into the emoji sentiment analysis model to obtain the image-text joint representation feature vector and the label feature vector of the emoji to be analyzed. The label feature vectors of the emoji to be analyzed with different labels are respectively calculated with the image-text joint representation feature vector of the emoji to be analyzed. The label result with the highest probability is output as the sentiment analysis result and intent detection result of the emoji to be analyzed.

[0038] This invention proposes a contrastive learning-based method for classifying emojis with metaphorical information. It considers the rich metaphorical information within emojis, making the textual information describing the emotional system of emojis more complete. By fusing emoji image features and text features, it identifies the feature relationships between the two, making it easier to uncover the emotional information contained within the emojis. Using the idea of ​​contrastive learning, it makes the matching pattern between label features and image-text joint information more intuitive and easier to understand the logic behind it, thus improving the accuracy in emoji emotion classification and intent detection classification. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a contrastive learning-based method for classifying emojis with metaphorical information, as shown in Example 1. Detailed Implementation

[0040] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0041] Example 1

[0042] This embodiment provides a method for classifying emojis with metaphorical information based on contrastive learning, such as... Figure 1 As shown, it includes the following steps:

[0043] S1. Extract valid data from the dataset and preprocess it to obtain an emoji dataset that integrates text information, Chinese and English emoji images, and label information. Divide the emoji dataset into training set, validation set, and test set. Valid data includes Chinese and English emoji images, emoji text information, emoji metaphor information, emoji sentiment information, and emoji intent information.

[0044] The metaphorical information in emojis includes source metaphorical information and target metaphorical information; the emotional information in emojis includes happiness, friendship, anger, sadness, fear, disgust, and surprise; and the intentional information in emojis includes interaction, expression, entertainment, offense, and others.

[0045] Preprocessing includes segmenting the text information and metaphorical information of the emoji using the segmentation flag [SEP] to obtain integrated text information; and converting the emotional information and intent information of the emoji into tag information, which includes sentences describing the emotional information and sentences describing the intent information of the emoji.

[0046] S2. Input the integrated text information from the emoji dataset obtained in step S1 into the text encoder to obtain the emoji text feature representation; input the Chinese and English emoji images into the image encoder to obtain the emoji image feature representation; input the label information into the text encoder to obtain the emoji label information feature representation.

[0047] In this embodiment, the text encoder selected is the DistilBERT encoder. The DistilBERT encoder is a model based on the BERT encoder after knowledge distillation. Compared with the BERT encoder, the DistilBERT encoder has fewer parameters and faster computation.

[0048] S201. Input the integrated text information into the tokenizer of the DistilBERT encoder to obtain the input number and attention flag of the corresponding text. The input number represents the result of mapping each word vector to a number after the integrated text information is segmented by the tokenizer. The attention flag uses the numbers 1 and 0 to represent different information. 1 represents the information that needs attention in the input number, and 0 represents the information that does not need attention in the input number.

[0049] S202. Input the input number and attention flag obtained in step S201 into the pre-trained DistilBERT encoder, and select the output vector corresponding to the special symbol [CLS] inserted before the integrated text information to represent the semantic representation of the integrated text information, so as to obtain a 768-dimensional emoji text feature representation;

[0050] S203. Input the Chinese and English emoticon images into cv2.imread for reading, convert the reading results to RGB format and input them into the albumentations image data augmentation library for reconstruction and normalization to obtain a tensor with a (3,224,224) structure.

[0051] S204. Input the tensor with the (3,224,224) structure obtained in step S203 into the pre-trained ResNet50 image encoder to obtain a 2048-dimensional emoji image feature representation;

[0052] S205. Input the tag information into the tokenizer of the DistilBERT encoder, and repeat steps S201-S202 to obtain a 768-dimensional feature representation of the emoji tag information.

[0053] S3. Using the text feature representation, image feature representation, and tag information feature representation of the emoticon obtained in step S2, calculate the image-text joint representation feature vector and the tag feature vector;

[0054] S301. The 768-dimensional emoji text feature representation obtained in step S202 is passed through a fully connected layer to transform the input dimension of 768 dimensions into an output dimension of 256 dimensions of emoji text vector. The emoji text vector is then passed through a GELU activation function and a fully connected layer with an output dimension of 256 dimensions. The result is then passed through a dropout_rate=0.20 blocking feature detector to obtain an emoji text blocking feature vector. The emoji text blocking feature vector is added to the emoji text vector. The result of the addition operation is normalized by LayerNorm to obtain the text features.

[0055] The 2048-dimensional emoji image feature representation obtained in step S204 is passed through a fully connected layer to transform the input dimension of 768 dimensions into an output dimension of 256 dimensions of emoji image vectors. The emoji image vectors are then passed through a GELU activation function and a fully connected layer with an output dimension of 256 dimensions. The result is then passed through a dropout_rate=0.20 blocking feature detector to obtain the emoji image blocking feature vector. The emoji image blocking feature vector is added to the emoji image vector, and the result of the addition operation is normalized by LayerNorm to obtain the image features.

[0056] S302. Add the image features and text features obtained in step S301 to obtain the image-text joint representation feature vector it. f ;

[0057] S303. The 768-dimensional emoji tag information feature representation obtained in step S305 is passed through a fully connected layer to transform the input dimension of 768 dimensions into an output dimension of 256 dimensions, resulting in the tag feature vector l. f .

[0058] S4. Based on the image-text joint representation feature vector and label feature vector obtained in step S3, construct an emoji sentiment analysis training model with metaphorical information based on contrastive learning;

[0059] S401. Calculate the cosine similarity between the image-text joint representation feature vector and the label feature vector to obtain the cosine similarity matrix logits, as shown in formula (1):

[0060] logits=np.dot(it f ,l f ) np.exp(t) (1)

[0061] Where t is a learnable parameter;

[0062] The diagonal elements of the cosine similarity matrix represent the calculation results of the image-text joint representation feature vector and the corresponding label feature vector, while the remaining elements of the cosine similarity matrix represent the calculation results of the image-text joint representation feature vector and the non-corresponding label feature vector.

[0063] S402. Construct a loss function based on the idea of ​​contrastive learning, as shown in formula (2):

[0064] (2)

[0065] Where T represents transpose. To calculate the cross-entropy loss, the label is calculated as shown in formula (3):

[0066] (3)

[0067] By maximizing the diagonal elements of the cosine similarity matrix and minimizing the remaining elements of the cosine similarity matrix using a loss function, we obtain a sentiment analysis model for emojis with metaphorical information based on contrastive learning.

[0068] S5. Use the training set obtained in step S1 to train the emoji sentiment analysis training model constructed in step S4, and save the trained emoji sentiment analysis training model.

[0069] S6. Use the validation set and test set obtained in step S1 to validate and test the emoji sentiment analysis training model trained in step S5, and obtain the emoji sentiment analysis model.

[0070] S7. Input the emoji to be analyzed into the emoji sentiment analysis model obtained in step S6 to obtain the sentiment classification result and intent detection classification result of the emoji to be analyzed.

[0071] The emoji to be analyzed is input into the emoji sentiment analysis model to obtain the image-text joint representation feature vector and the label feature vector of the emoji to be analyzed. The label feature vectors of the emoji to be analyzed with different labels are calculated with the image-text joint representation feature vector of the emoji to be analyzed. The label result with the highest probability is output as the sentiment analysis result and intent detection result of the emoji to be analyzed.

[0072] Example 2

[0073] This embodiment provides a method for classifying emojis with metaphorical information based on contrastive learning, using the MET-Meme dataset, and includes the following steps:

[0074] Step 1: Extract valid data from the MET-Meme dataset and preprocess it, then divide it into training set, validation set and test set, with a ratio of 8:1:1.

[0075] The MET-Meme dataset includes Chinese and English emoji datasets. This embodiment uses the English emoji dataset for experiments. Emoji images are stored in one folder, while the image name, text information, metaphor information, sentiment information, intent information, source domain text information, and target domain text information are stored in CSV files. The training set contains 3200 data points, the validation set contains 400 data points, and the test set contains 400 data points.

[0076] The specific preprocessing of the data includes:

[0077] (1) Store the data from the MET-Meme dataset csv file in a pandas dataframe structure;

[0078] (2) Separate the text information and metaphorical information of the emoji using the separator [SEP]. The resulting integrated text information can be represented in the following format:

[0079] df['text'] + ' [SEP] ' + df['source_text'] + ' [SEP] ' + df['target_text']

[0080] (3) The emotional information and intent information of the emojis are converted into tag information. The tag information includes sentences describing the emotional information of the emojis and sentences describing the intent information of the emojis. The specific templates for the sentences describing the emotional information and the sentences describing the intent information of the emojis are as follows:

[0081] Template for sentences describing the emotional message of an emoji: 'a meme with [sentiment label]emotion'

[0082] Template for sentences describing the intent of an emoji: 'a meme with [intention label]intention'

[0083] Step 2: Obtain the text feature representation, image feature representation, and sentence-converted emoji tag information feature representation, respectively. This includes the following steps:

[0084] (1) Calculate the text feature representation of the emoji:

[0085] (a) After preprocessing in step one, all text information has the following text format:

[0086] df['text'] + ' [SEP] ' + df['source_text'] + ' [SEP] ' + df['target_text']

[0087] The integrated text information in this text format is input into the tokenizer of the DistilBERT encoder. First, the tokenizer will segment the input sentence into tokens. On this basis, a special token [CLS] is added before the sentence and a special token [SEP] is added at the end of the sentence. After obtaining the tokens, the tokenizer will replace each token with the ID in the embeddings table built into the model.

[0088] (b) At the same time as step (a) is completed, the tokenizer returns the input number input_ids and the attention flag attention_mask. The input number input_ids is the ID value of the embedding represented by the sentence information mentioned in step (a). The attention flag attention_mask uses the numbers 1 and 0 to represent different information. Since the length of each input sentence is different, in order to facilitate processing, 0 and 1 are used to represent whether the information at the corresponding position needs to be processed. 1 represents the information that needs attention in the input number, and 0 represents the information that does not need attention in the input number.

[0089] (c) Input the input number input_ids and attention flag attention_mask obtained in step (b) into the pre-trained DistilBERT encoder. Since the subsequent task is to obtain the feature vectors required for classification, the output of the model related to the [CLS] token is used as the semantic representation of the text information. The final result is a 768-dimensional emoji text feature representation.

[0090] (2) Calculate the feature representation of emoji images:

[0091] (a) For emoji images, the images are read using cv2.imread and converted to the common RGB format using cv2.cvtColor. Then, the image data augmentation library is used to resize and normalize the read images to obtain an ndarray array with a (224,224,3) structure. Finally, it is converted into a tensor with a (3,224,224) structure.

[0092] (b) The tensor with the structure (3,224,224) obtained in step (a) is used as input and fed into the pre-trained ResNet50 image encoder using the average pooling mode. After passing through the last layer of adaptive average pooling, the final result is a 2048-dimensional feature representation of the emoji image.

[0093] (3) Calculate the feature representation of emoji tag information:

[0094] Following the method in step (1), the feature representation of the tag information after conversion into a sentence is calculated in the same way, and the final result is a 768-dimensional feature representation of the emoji tag information.

[0095] Step 3: Calculate the image-text joint representation feature vector and the label feature vector, which includes the following steps:

[0096] (1) Calculate the eigenvectors of the combined text-image representation:

[0097] (a) In step two, the 768-dimensional emoji text feature representation t has been obtained. emb And 2048-dimensional emoji image feature representation i emb The text and image feature representations of emojis are processed through a projection head, projected onto a new space, and a joint text-image representation feature vector is obtained. The calculation formula is: it f =g(t emb )+g(i emb ), where it f The feature vector is a joint representation of text and image. The calculation process for the projection head is as follows: The 768-dimensional emoji text feature representation is passed through a fully connected layer, transforming the input dimension of 768 into an output dimension of 256 emoji text vectors. These emoji text vectors are then passed through a GELU activation function and another fully connected layer with an output dimension of 256. The result is then passed through a dropout-rate=0.20 blocking feature detector to obtain the emoji text blocking feature vector. This blocking feature vector is then added to the original emoji text vector. The result of this addition is normalized using LayerNorm to obtain the text feature t. f , t f =g(t emb );

[0098] The 2048-dimensional emoji image feature representation is passed through a fully connected layer, transforming the input dimension of 768 into an output dimension of 256 emoji image vectors. These vectors are then processed through a GELU activation function and another fully connected layer with an output dimension of 256. The result is then passed through a dropout-rate-0.20 blocking feature detector to obtain the emoji image blocking feature vector. This blocking feature vector is then added to the original emoji image vector. The result of this addition is normalized using LayerNorm to obtain the image feature i. f i f =g(i emb );

[0099] (b) The obtained text features t f and image features i f Perform the addition operation to obtain the image-text joint representation feature vector it. f .

[0100] (2) Calculate the label feature vector:

[0101] The 768-dimensional emoji tag information features are passed through a fully connected layer with an input dimension of 768 and an output dimension of 256 to obtain the tag feature vector l. f .

[0102] Step 4: Construct a training model for sentiment analysis of emojis with metaphorical information based on contrastive learning, which includes the following steps:

[0103] (1) After the calculation in step three, the image-text joint representation feature vector it with shape (batch_size, 256) is obtained. f With a label feature vector l of shape (batch_size, 256) f Calculate the cosine similarity between the two, as shown in formula (4):

[0104] logits=np.dot(it f ,l f ) np.exp(t) (4)

[0105] Where t is a learnable parameter, a cosine similarity matrix of shape (batch_size, batch_size) can be obtained through the above calculation. The diagonal elements in the cosine similarity matrix represent the calculation results of the image-text joint representation feature vector and the corresponding label feature vector, while the remaining elements are the calculation results of the image-text joint representation feature vector and the non-corresponding label feature vector.

[0106] (2) Based on the idea of ​​contrastive learning, the training objective is to make the diagonal elements of the above matrix positive samples and the remaining elements negative samples. The loss function is constructed using this idea and calculated as shown in formula (5):

[0107] (5)

[0108] in, The standard method for calculating cross-entropy loss is used, and the label is calculated as shown in formula (6):

[0109] (6)

[0110] By maximizing the diagonal elements of the cosine similarity matrix and minimizing the remaining elements of the cosine similarity matrix using a loss function, we obtain a sentiment analysis model for emojis with metaphorical information based on contrastive learning.

[0111] Step 5: Use the training set obtained in Step 1 to train the emoji sentiment analysis training model with metaphorical information based on contrastive learning constructed in Step 4, and save the trained emoji sentiment analysis training model with metaphorical information based on contrastive learning.

[0112] Step 6: Use the validation set and test set obtained in Step 1 to validate and test the emoji sentiment analysis training model trained in Step 5, and obtain the emoji sentiment analysis model.

[0113] Step 7: Input the emoji to be analyzed into the emoji sentiment analysis model obtained in Step 6 to obtain the sentiment classification results and intent detection classification results of the emoji to be analyzed.

[0114] The training data was trained using an emoji sentiment analysis model, and the results were validated using experimental results on a test set. The CLMET emoji classification method based on contrastive learning with metaphorical information in this implementation was compared with existing baseline methods. The results are shown in Table 1.

[0115] Table 1 Comparison of experimental results for different algorithms

[0116]

[0117] As shown in the table, the contrastive learning-based emoji classification method CLMET, proposed in this implementation, achieves better results than existing baseline methods on the same dataset. In sentiment classification, CLMET achieves 0.0776 higher accuracy and 0.0511 higher recall than existing baseline methods. Furthermore, CLMET achieves a higher F1 score than existing baseline methods. The score was 0.0541 higher. In intent detection classification, the CLMET emoji classification method based on contrastive learning with metaphorical information in this implementation achieved an accuracy 0.0877 higher than the existing baseline method, a recall 0.025 higher, and an F1 score 0.032 higher. This indicates that the overall predictive ability of the CLMET emoji classification method based on contrastive learning with metaphorical information in this implementation is stronger than that of the existing baseline method.

[0118] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for classifying emojis with metaphorical information based on contrastive learning, characterized in that, Includes the following steps: S1. Extract valid data from the dataset and preprocess it to obtain an emoji dataset that integrates text information, Chinese and English emoji images, and tag information. Divide the emoji dataset into a training set, a validation set, and a test set. The valid data includes Chinese and English emoji images, emoji text information, emoji metaphor information, emoji sentiment information, and emoji intent information. S2. Input the integrated text information from the emoji dataset obtained in step S1 into the text encoder to obtain the emoji text feature representation; input the Chinese and English emoji images into the image encoder to obtain the emoji image feature representation; input the tag information into the text encoder to obtain the emoji tag information feature representation. S3. Using the text feature representation, image feature representation, and tag information feature representation of the emoticon obtained in step S2, calculate the image-text joint representation feature vector and the tag feature vector; S4. Based on the image-text joint representation feature vector and label feature vector obtained in step S3, construct an emoji sentiment analysis training model with metaphorical information based on contrastive learning; S5. Use the training set obtained in step S1 to train the emoji sentiment analysis training model constructed in step S4, and save the trained emoji sentiment analysis training model. S6. Use the validation set and test set obtained in step S1 to validate and test the emoji sentiment analysis training model trained in step S5, and obtain the emoji sentiment analysis model. S7. Input the emoji to be analyzed into the emoji sentiment analysis model obtained in step S6 to obtain the sentiment classification result and intent detection classification result of the emoji to be analyzed.

2. The method for classifying emojis with metaphorical information based on contrastive learning according to claim 1, characterized in that, In step S1, the metaphorical information of the emoticon includes source metaphorical information and target metaphorical information; the emotional information of the emoticon includes happiness, friendship, anger, sadness, fear, disgust, and surprise; and the intentional information of the emoticon includes interaction, expression, entertainment, offense, and others.

3. The method for classifying emojis with metaphorical information based on contrastive learning according to claim 2, characterized in that, In step S1, the preprocessing includes segmenting the emoji text information and emoji metaphor information using the segmentation flag [SEP], and obtaining integrated text information after segmentation; The emotional information and intent information of the emojis are converted into tag information, which includes sentences describing the emotional information of the emojis and sentences describing the intent information of the emojis.

4. The method for classifying emojis with metaphorical information based on contrastive learning according to claim 3, characterized in that, Step S2 includes: S201. Input the integrated text information into the tokenizer of the DistilBERT encoder to obtain the input number and attention flag of the corresponding text. The input number represents the result of mapping each word vector to a number after the integrated text information is segmented by the tokenizer. The attention flag uses the numbers 1 and 0 to represent different information. 1 represents information that needs attention in the input number, and 0 represents information that does not need attention in the input number. S202. Input the input number and attention flag obtained in step S201 into the pre-trained DistilBERT encoder, select the output vector corresponding to the special symbol [CLS] inserted before the integrated text information to represent the semantic representation of the integrated text information, and obtain a 768-dimensional emoji text feature representation; S203. Input the Chinese and English emoticon images into cv2.imread for reading, convert the reading result into RGB format and input it into the albumentations image data augmentation library for reconstruction and normalization to obtain a tensor with a (3,224,224) structure. S204. Input the tensor with the (3,224,224) structure obtained in step S203 into the pre-trained ResNet50 image encoder to obtain a 2048-dimensional feature representation of the emoji image; S205. Input the tag information into the word segmenter of the DistilBERT encoder, and repeat steps S201-S202 to obtain a 768-dimensional feature representation of the emoji tag information.

5. The method for classifying emojis with metaphorical information based on contrastive learning according to claim 4, characterized in that, Step S3 includes: S301. The 768-dimensional emoji text feature representation obtained in step S202 is passed through a fully connected layer to convert the input dimension of 768 dimensions into an output dimension of 256 dimensions of emoji text vector. The emoji text vector is then passed through a GELU activation function and a fully connected layer with an output dimension of 256 dimensions. The result is then passed through a dropout_rate=0.20 blocking feature detector to obtain an emoji text blocking feature vector. The emoji text blocking feature vector is then added to the emoji text vector. The result of the addition operation is normalized by LayerNorm to obtain the text feature. The 2048-dimensional emoji image feature representation obtained in step S204 is passed through a fully connected layer to transform the input dimension of 768 dimensions into an output dimension of 256 dimensions for the emoji image vector. The emoji image vector is then passed through a GELU activation function and a fully connected layer with an output dimension of 256 dimensions. The result is then passed through a dropout_rate=0.20 blocking feature detector to obtain the emoji image blocking feature vector. The emoji image blocking feature vector is then added to the emoji image vector. The result of the addition operation is normalized using LayerNorm to obtain the image features. S302. Add the image features and text features obtained in step S301 to obtain the image-text joint representation feature vector it. f ; S303. The 768-dimensional emoji tag information feature representation obtained in step S205 is passed through a fully connected layer to transform the input dimension of 768 dimensions into an output dimension of 256 dimensions, resulting in the tag feature vector l. f .

6. The method for classifying emojis with metaphorical information based on contrastive learning according to claim 5, characterized in that, Step S4 includes: S401. Calculate the cosine similarity between the image-text joint representation feature vector and the label feature vector to obtain the cosine similarity matrix logits, as shown in formula (1): logits=np.dot(it f ,l f ) np.exp(t) (1) Where t is a learnable parameter, it f The image-text joint representation feature vector, l f Represents the label feature vector; The diagonal elements of the cosine similarity matrix represent the calculation results of the image-text joint representation feature vector and the corresponding tag feature vector, and the remaining elements of the cosine similarity matrix are the calculation results of the image-text joint representation feature vector and the non-corresponding tag feature vector. S402. Construct a loss function based on the idea of ​​contrastive learning, as shown in formula (2): （2） Where T represents transpose. To calculate the cross-entropy loss, the label is calculated as shown in formula (3): （3） By maximizing the diagonal elements of the cosine similarity matrix and minimizing the remaining elements of the cosine similarity matrix using the loss function, a sentiment analysis model for emojis with metaphorical information based on contrastive learning is obtained.

7. The method for classifying emojis with metaphorical information based on contrastive learning according to claim 6, characterized in that, In step S7, the emoticon to be analyzed is input into the emoticon sentiment analysis model to obtain the image-text joint representation feature vector and the label feature vector of the emoticon to be analyzed. The label feature vectors of the emoticon to be analyzed with different labels are calculated with the image-text joint representation feature vector of the emoticon to be analyzed. The label result with the highest probability is output as the sentiment analysis result and intent detection result of the emoticon to be analyzed.

Images