Emoticon sentiment recognition method based on large model multi-modal multi-granularity feature fusion

Abstract

The application provides a large model-based multi-modal multi-granularity feature fusion emoticon emotion recognition method, which comprises the following steps: inputting a picture-text data pair into a CLIP model; constructing a similarity graph representing fine-grained picture-text consistency based on cosine similarity; screening partial image block features with the highest similarity to text features; fusing image global features and image block features corresponding to sequence indexes to obtain multi-granularity image feature representation; adjusting and freezing the CLIP model; obtaining a to-be-recognized emoticon and repeatedly performing image fusion processing via the current frozen CLIP model image encoder to obtain to-be-recognized fusion image features, which are embedded into a preset prompt template and input into an LLM model for fine-tuning and emotion category prediction; adjusting the LLM model and performing iteration until the LLM model meets preset requirements in emotion recognition. Through two-stage fine-tuning, the application realizes accurate recognition and classification of emoticon emotions.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a method for emoji emotion recognition based on multimodal and multigranular feature fusion using a large model. Background Technology

[0002] With the continuous advancement of artificial intelligence (AI) technology, its applications on the internet have become increasingly widespread and profound. Especially in emotional expression and online communication, AI has demonstrated unprecedented potential. Developers can optimize online communities by judging the emotional information in users' emojis. This invention relates to an AI-based emoji emotion recognition method, aiming to automatically analyze emojis and accurately identify the specific emotions they express. This method uses deep learning and pattern recognition technologies to analyze image content and annotate emotions, thereby achieving intelligent recognition of emoji emotions. The emoji emotion recognition technology of this invention provides new tools and perspectives for research and development in fields such as emotion analysis, user behavior research, and personalized recommendation systems. This technological advancement helps to understand users' emotional responses and behavioral patterns more deeply, thus playing a key role in improving user experience and interaction quality.

[0003] Sentiment recognition, as an important direction in Natural Language Processing (NLP), has expanded from single-text analysis to the interpretation of multimodal data, including images, videos, and audio. Among the vast array of multimodal data, emojis, as a widely popular element of online communication, offer new research objects and challenges for sentiment analysis due to their rich and intuitive emotional expression. Emojis typically contain visual elements and underlying textual information, requiring sentiment recognition technology to not only understand the image content but also combine it with related textual descriptions to achieve a deeper level of emotional understanding.

[0004] Furthermore, the development of multimodal emotion recognition technology reflects the evolution from single-modal to multimodal interaction. For example, the emergence of multimodal pre-trained models such as the CLIP model provides powerful tools for joint understanding and sentiment analysis of images and text. These models can learn the correlation between images and text, providing new solutions for emotion recognition of emojis. However, multimodal emotion recognition also faces a series of challenges. Semantic alignment between different modalities, information redundancy or complementarity between modalities, and differences in emotional expression across different cultural backgrounds are all problems that need to be addressed in this field. In addition, the high cost and difficulty in obtaining multimodal data annotation also limit the training and optimization of models. To overcome these challenges, researchers are exploring more advanced model architectures, such as attention mechanisms and graph neural networks, to better integrate and learn interactive information between different modalities. At the same time, interdisciplinary collaborations, such as the participation of psychology, sociology, and cognitive science, also contribute to a deeper understanding of the nature of emotions and promote the development of emotion recognition technology.

[0005] In today's social networks, users express their emotions and attitudes through various means such as text, images, and emojis. Among these, emojis, due to their concise and intuitive nature, are widely used on social media, often quickly conveying the sender's emotional state. However, traditional text sentiment analysis techniques are difficult to apply directly to emojis because the visual elements and underlying textual information within emojis need to be analyzed together to accurately identify their emotional inclinations. Secondly, a series of emojis often use a fixed character (such as a virtual cartoon character) as the main subject to express emotions, and these emojis are highly similar in character and style. Therefore, it is necessary to distinguish the emotions they convey through subtle differences in details. Furthermore, the diversity and complex components of emojis make this task even more difficult. To address this challenge, this invention proposes an emoji sentiment recognition system based on a large-scale model and multimodal feature fusion. This method can not only analyze the complex visual elements in emojis and capture their subtle differences, but also combine them with related textual information, using deep learning and pattern recognition techniques to achieve accurate identification of emoji emotions. This approach allows for a more comprehensive understanding of users' emotional expressions in social interactions, providing more accurate sentiment analysis tools for fields such as social media analytics, user behavior research, and personalized recommendation systems.

[0006] However, there are some limitations in existing technologies such as "multimodal local reattention networks" and "Sticker CLIP":

[0007] 1. Monomodal Dependence: Existing techniques such as "multimodal local reattention networks" mainly focus on integrating global and local features of emojis through local attention mechanisms. However, this approach may rely too heavily on information from a single modality (image) and fail to fully utilize the potential of multimodal data.

[0008] 2. Depth of emotion recognition: Although StickerCLIP improves emotion recognition performance through large-scale image-text alignment, it may have limitations in capturing and understanding subtle emotional differences in emojis, especially when the emotional expression is complex or subtle. Summary of the Invention

[0009] The technical problem to be solved by this invention is how to make full use of the consistency of multimodal data to accurately capture and understand the subtle emotional differences in emojis and improve the accuracy of emotion recognition. In view of this, this invention provides an emoji emotion recognition method based on multimodal and multigranular feature fusion of a large model.

[0010] The technical solution adopted in this invention is a method for emoji emotion recognition based on multimodal and multigranular feature fusion using a large model, comprising:

[0011] Step 1: Obtain the image and text data pairs extracted from the emojis and input them into the CLIP model to obtain global image features, image patch features, and text features;

[0012] Step 2: Based on the cosine similarity between the text features and the features of each image block, construct a similarity map representing fine-grained text-image consistency;

[0013] Step 3: Based on the similarity map, use the configured threshold to filter the image patch features with the highest similarity to the text features, and obtain the sequence index of the image patch;

[0014] Step 4: Fuse the global features of the image with the image patch features corresponding to the sequence index to obtain multi-granularity fused image features;

[0015] Step 5: Based on the multi-granularity fused image features, the CLIP model is adjusted and frozen by using a fully connected layer as the sentiment prediction head and the cross-entropy loss function as the sentiment classification loss.

[0016] Step 6: Obtain the emoji to be recognized and pass it through the currently frozen CLIP model image encoder. Repeat steps 1 to 4 to obtain the fused image features to be recognized.

[0017] Step 7: Embed the features of the image to be identified into a preset prompt template to form an input structure;

[0018] Step 8: Input the input structure into the LLM model for fine-tuning and predict the sentiment category;

[0019] Step 9: Adjust the LLM model and iterate it based on the emotion category prediction results until the LLM model meets the preset requirements in emotion recognition.

[0020] In one embodiment, the method further includes:

[0021] For emoji images in the acquired emoji dataset that do not have accompanying text annotations, use the pre-trained BLIP-2 model to generate descriptive text for these images to augment the image-text data pairs in the dataset.

[0022] In one implementation, in step 1,

[0023] The image and text data output text features F via multiple ViT modules of the encoder. text Image features F img The hidden layer image features are represented as follows:

[0024]

[0025] in, This represents the global feature vector, and L represents the number of ViT modules;

[0026] Image patch features are:

[0027]

[0028] In one implementation, step 2 includes:

[0029]

[0030] Where sim is the fine-grained similarity graph, · represents the vector inner product operation, and ||·||2 represents the Euclidean norm of the vector.

[0031] In one implementation, in step 4, the multi-granularity fused image features are determined by the following formula:

[0032]

[0033] in, represents the global feature vector output by layer L-1, and k represents the k image patch vectors with the highest cosine similarity selected.

[0034] In one implementation, step 5 includes:

[0035] The probability distribution for each emotion category is determined as follows:

[0036]

[0037] Where W is the weight matrix of the linear layer, C represents the total number of sentiment categories, and P i This represents the probability that the model predicts the i-th sentiment category;

[0038] The difference between the model output probability distribution P and the true label distribution Y is calculated using the cross-entropy loss function, and the CLIP is fine-tuned by minimizing the sentiment classification loss.

[0039] The cross-entropy loss function L is shown in the following equation:

[0040]

[0041] The CLIP model is adjusted by minimizing the cross-entropy loss function to map text-image pair features to the correct sentiment category.

[0042] Freeze the adjusted CLIP model.

[0043] In one implementation, the preset prompt template in step 7 includes: the fused image features to be identified, the corresponding text, and a specific sentiment prediction prompt.

[0044] In one implementation, in step 9, based on the predicted sentiment category, the model is iteratively improved by adjusting the loss function weights and hyperparameters of the LLM model until it achieves optimal performance on the sentiment recognition task.

[0045] Another aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the emoji emotion recognition method based on a large model as described in any of the preceding claims.

[0046] Another aspect of the present invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the emoji emotion recognition method based on a large model as described in any of the preceding claims.

[0047] Compared with the prior art, the present invention has at least the following advantages:

[0048] This application provides a multimodal feature fusion emoji emotion recognition system based on a large model. Through two stages of fine-tuning, it achieves accurate identification and classification of emoji emotions. In the first stage, a pre-trained BLIP-2 model is used to generate descriptive text for unlabeled emojis, and the CLIP model is fine-tuned to learn image feature representations and enhance the ability to select key local features. These feature representations maintain consistency with text features at the content level and emphasize subtle differences between emoji elements. In the second stage, the parameters of the CLIP model are frozen, and multi-granular image representations and text are embedded in a structured preset template. The large pre-trained language model is then fine-tuned to predict emotion categories, further improving the model's emotion recognition capability. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the process of the emoji emotion recognition method based on large model multimodal multigranularity feature fusion according to an embodiment of the present invention;

[0050] Figure 2 This is a logical schematic diagram of the emoji emotion recognition method based on a large model and multimodal multigranularity feature fusion according to an embodiment of the present invention.

[0051] Figure 3 This is another flowchart illustrating the emoji emotion recognition method based on a large model and multimodal multigranularity feature fusion according to an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0053] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.

[0054] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms (e.g., those defined in common dictionaries) shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art and shall not be interpreted in an idealized or overly formal sense unless expressly so specified herein.

[0055] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0056] The abbreviations and key terms used in this article are defined as follows:

[0057] OCR: Optical Character Recognition;

[0058] BLIP-2: Bootstrapping Language-Image Pretraining, a multimodal image-text pretraining model proposed by Salesforce in 2023;

[0059] CLIP: Contrastive Language-Image Pre-Training, utilizes contrastive learning to pre-train large-scale language-image models;

[0060] ViT: Vision Transformer, a model proposed by the Google team in 2020 that applies the Transformer to image classification;

[0061] LLM: Large Language Model;

[0062] ReLU: Rectified LinearUnit, a linear rectified function, a type of activation function;

[0063] BP: Back Propagation;

[0064] Adam: A method for stochastic Optimization.

[0065] This invention provides an embodiment of a multimodal, multi-granularity feature fusion emoji emotion recognition method based on a large model, such as... Figure 1 As shown, it includes:

[0066] Step 1: Obtain the image and text data pairs extracted from the emojis and input them into the CLIP model to obtain global image features, image patch features, and text features;

[0067] Step 2: Based on the cosine similarity between the text features and the features of each image block, construct a similarity map representing fine-grained text-image consistency;

[0068] Step 3: Based on the similarity map, use the configured threshold to filter the image patch features with the highest similarity to the text features, and obtain the sequence index of the image patch;

[0069] Step 4: Fuse the global features of the image with the image patch features corresponding to the sequence index to obtain multi-granularity fused image features;

[0070] Step 5: Based on the multi-granularity fused image features, the CLIP model is adjusted and frozen by using a fully connected layer as the sentiment prediction head and the cross-entropy loss function as the sentiment classification loss.

[0071] Step 6: Obtain the emoji to be recognized and pass it through the currently frozen CLIP model image encoder. Repeat steps 1 to 4 to obtain the fused image features to be recognized.

[0072] Step 7: Embed the features of the image to be identified into a preset prompt template to form an input structure;

[0073] Step 8: Input the input structure into the LLM model for fine-tuning and predict the sentiment category;

[0074] Step 9: Adjust the LLM model and iterate it based on the emotion category prediction results until the LLM model meets the preset requirements in emotion recognition.

[0075] The method provided in this embodiment will be described in detail step by step below.

[0076] As attached Figure 2 , 3 As shown, this application provides a multimodal feature fusion emoji emotion recognition system based on a large model. The specific implementation scheme is divided into two stages. The first stage includes:

[0077] We selected emoji images without accompanying text annotations from the dataset and used a pre-trained BLIP-2 model to generate descriptive text for these images to enrich the text information in the dataset, which facilitates subsequent fine-grained image-text alignment work.

[0078] Step 1: Input the image-text pair data into the CLIP model to obtain image patch encoding and text encoding. The model input can be represented as:

[0079] Input = {V, T, Y}

[0080] Where V represents the emoji image, T represents the text extracted from the dataset by OCR or the descriptive text generated by the BLIP-2 model in step one, and Y represents the sentiment category label of the emoji.

[0081] It should be noted that the CLIP model's image encoder consists of multiple Transformer modules. Each module contains a multi-head attention layer (MSA), an MLP layer, and two residual connections. LN represents layer normalization, and the calculation process is as follows:

[0082]

[0083] The data is processed by multiple ViT modules of the encoder to output text features F. text Image features F img The hidden layer image features are represented as follows:

[0084]

[0085] in represents the global feature vector, and L represents the number of ViT modules.

[0086] Image patch vectors can be represented as:

[0087]

[0088] In some implementations, emoji images without accompanying text annotations can be filtered from the dataset before input, and descriptive text can be generated for these images using a pre-trained BLIP-2 model to augment the text information in the dataset, facilitating subsequent fine-grained image-text alignment.

[0089] Step 2: Calculate the cosine similarity between text features and each image patch vector to construct a similarity map reflecting fine-grained image-text consistency, specifically represented as follows:

[0090]

[0091] Among them, F text and F i_t As obtained from the previous step, "·" represents the vector inner product operation, and ||·||2 represents the Euclidean norm (L2 norm) of the vector;

[0092] Step 3: Based on the cosine similarity map obtained in Step 2, select the image patches with the highest similarity to the text features and obtain their sequence index;

[0093] Step 4: Based on the selected index, fuse the global image features with the image patch features selected in Step 4;

[0094] Image patches are selected based on their indices, and the global feature vector is concatenated with the image patch vector. This concatenation is then input into the last Transformer layer, resulting in the fused image feature representation:

[0095]

[0096] in represents the global feature vector output by layer L-1, and k represents the k image patch vectors with the highest cosine similarity selected.

[0097] By fusing global feature vectors and image patch vectors, not only is global information preserved and multi-granular image feature representation obtained, but the last layer is also forced to focus the Transformer on key regions, reducing interference from background and other information.

[0098] Step 5: Using a fully connected layer as the sentiment prediction head, the fused feature representation after the Transformer layer will be passed through a linear layer (also called a fully connected layer), and then normalized through a softmax layer to generate the probability distribution of each sentiment category, calculated as follows:

[0099]

[0100] Where W is the weight matrix of the linear layer, C represents the total number of sentiment categories, and P i This represents the probability that the model predicts the i-th sentiment category.

[0101] Optionally, the CLIP can be fine-tuned by calculating the difference between the model's output probability distribution P and the true label distribution Y using the cross-entropy loss function, minimizing the sentiment classification loss. During fine-tuning, the model learns to map image and text features to the correct sentiment category. The loss function L is expressed as:

[0102]

[0103] By minimizing the cross-entropy loss function L, the model learns to map image and text features to the correct sentiment category, thereby optimizing the sentiment classification task.

[0104] The primary goal of the first-stage training strategy is to fully leverage the information consistency among multimodal data to optimize image feature representation. Multi-granularity image fusion features integrate global information and subtle differences from emojis, serving as a crucial input component for the second stage.

[0105] The second stage employed LLM for sentiment inference, leveraging its powerful language processing capabilities and prior knowledge to fine-tune the model and improve its ability to identify sentiment-related information. Specific implementation included:

[0106] Step 6: Pass the emoji image through the frozen CLIP model image encoder, and repeat steps 1 to 4 of the previous stage to obtain image fusion features.

[0107] Step 7: Embed the fused image features into the designed prompt template to form the final input structure.

[0108] Furthermore, this application embodiment designs a structured input template, which can be represented as: "image fusion features + text + specific sentiment prediction prompts".

[0109] The components are explained below:

[0110] Image fusion features: These are multi-granularity fusion representations of the global and key local features of the image obtained through step 6. They are vector representations extracted by the CLIP model and processed by the Transformer layer.

[0111] Text: Text data can be the original caption of an emoji or descriptive text generated by the BLIP-2 model.

[0112] Sentiment prediction cue: This is a specifically designed cue template used to explicitly inform the model that the next task is sentiment prediction. For example, using "ithas[Mask]emotion" is a viable template, where [Mask] is a placeholder representing the sentiment category to be predicted, offering flexibility and interpretability. This allows the model to learn representations of different sentiment categories during training, helping it generalize to new or unseen sentiment categories;

[0113] Step 8: Input the input structure obtained in the previous step into the LLM for fine-tuning. During the fine-tuning process, the model is guided to learn and predict the sentiment category through the [Mask] part of the prompt template, and the cross-entropy loss function is still used in the process;

[0114] Step 9: Based on the sentiment prediction results, adjust the weights and hyperparameters of the LLM loss function and iteratively improve the model until it achieves optimal performance on the sentiment recognition task.

[0115] Furthermore, in both stages of fine-tuning in this embodiment, the backpropagation (BP) algorithm is used to calculate the gradient of the model parameters, and the Adam optimizer is used to update the model weights. Through iterative fine-tuning in these two stages, the model weights are gradually updated to minimize the loss function of the sentiment classification task.

[0116] like Figure 3As shown in the illustration, this application provides a multimodal feature fusion emoji emotion recognition system based on a large model. Through two stages of fine-tuning, it achieves accurate identification and classification of emoji emotions. In the first stage, a pre-trained BLIP-2 model is used to generate descriptive text for unlabeled emojis, and the CLIP model is fine-tuned to learn image feature representations and enhance the ability to select key local features. These multi-granular feature representations maintain consistency with text features at the content level and emphasize the subtle differences between emoji elements. In the second stage, the parameters of the CLIP model are frozen, and the multi-granular image representations and text are embedded in a structured preset template. The large pre-trained language model is then fine-tuned to predict emotion categories, further improving the model's emotion recognition ability. This application embodiment achieves certain performance improvements in the accuracy of emotion recognition, the model's adaptability in different contextual environments, and the ability to understand complex emotional expressions.

[0117] In summary, compared with the prior art, the present invention has at least the following advantages:

[0118] This application provides a multimodal feature fusion emoji emotion recognition system based on a large model. Through two stages of fine-tuning, it achieves accurate identification and classification of emoji emotions. In the first stage, a pre-trained BLIP-2 model is used to generate descriptive text for unlabeled emojis, and the CLIP model is fine-tuned to learn image feature representations and enhance the ability to select key local features. These feature representations maintain consistency with text features at the content level and emphasize subtle differences between emoji elements. In the second stage, the parameters of the CLIP model are frozen, and a large pre-trained language model is fine-tuned to predict emotion categories, further improving the model's emotion recognition ability. The key innovations of this application include:

[0119] Fine-grained image-text consistency: By calculating the cosine similarity between text features and image patch vectors, a similarity map is constructed, revealing the consistency relationship between each local feature patch in the text and the image. This fine-grained consistency analysis provides a basis for accurately selecting local features that highly match the emotional content of the text, ensuring the deep fusion and accurate correspondence of image and text information, and effectively addressing the issue of similarity between emoji subjects.

[0120] Multimodal feature fusion: The multimodal feature fusion strategy in this application not only involves combining global image features with key local image patch features to form a complete multi-granularity feature representation, but also includes designing a unified template to input into a large language model, further improving the utilization efficiency of multimodal information. This design allows the model to process image and text features simultaneously, ensuring the integrity and complementarity of information;

[0121] Large-scale language model fine-tuning: In the second stage, the frozen CLIP model is used to generate fused image feature representations, taking into account both global information and subtle differences in emojis, and the LLM is fine-tuned. The [Mask] part of the prompt template guides the model to learn and predict sentiment categories, further optimizing the performance of sentiment recognition.

[0122] The embodiments of this application have achieved certain performance improvements in the accuracy of emotion recognition, the adaptability of the model in different contexts, and the ability to understand complex emotional expressions.

[0123] A second embodiment of the present invention provides an electronic device, such as... Figure 4 As shown, it can be understood as a physical device, including a processor and a memory storing processor-executable instructions. When the instructions are executed by the processor, the following operations are performed:

[0124] Step 1: Obtain the image and text data pairs extracted from the emojis and input them into the CLIP model to obtain global image features, image patch features, and text features;

[0125] Step 2: Based on the cosine similarity between the text features and the features of each image block, construct a similarity map representing fine-grained text-image consistency;

[0126] Step 3: Based on the similarity map, use the configured threshold to filter the image patch features with the highest similarity to the text features, and obtain the sequence index of the image patch;

[0127] Step 4: Fuse the global features of the image with the image patch features corresponding to the sequence index to obtain multi-granularity fused image features;

[0128] Step 5: Based on the multi-granularity fused image features, the CLIP model is adjusted and frozen by using a fully connected layer as the sentiment prediction head and the cross-entropy loss function as the sentiment classification loss.

[0129] Step 6: Obtain the emoji to be recognized and pass it through the currently frozen CLIP model image encoder. Repeat steps 1 to 4 to obtain the fused image features to be recognized.

[0130] Step 7: Embed the features of the image to be identified into a preset prompt template to form an input structure;

[0131] Step 8: Input the input structure into the LLM model for fine-tuning and predict the sentiment category;

[0132] Step 9: Adjust the LLM model and iterate it based on the emotion category prediction results until the LLM model meets the preset requirements in emotion recognition.

[0133] In the third embodiment of the present invention, the process of the multimodal, multi-granularity feature fusion emoji emotion recognition method based on a large model is the same as that of the first and second embodiments. The difference lies in the engineering implementation: this embodiment can be implemented using software plus necessary general-purpose hardware platforms. While hardware implementation is also possible, the former is often a better approach. Based on this understanding, the method of the present invention can be embodied in the form of a computer software product stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including several instructions to cause a device to execute the method described in the embodiments of the present invention.

[0134] Through the description of specific embodiments, a more in-depth and specific understanding should be gained of the technical means and effects adopted by the present invention to achieve the intended purpose. However, the accompanying drawings are only provided for reference and illustration and are not intended to limit the present invention.

Claims

1. A method for emoji emotion recognition based on multimodal and multigranular feature fusion using a large model, characterized in that, include: Step 1: Obtain the image and text data pairs extracted from the emojis and input them into the CLIP model to obtain global image features, image patch features, and text features; Step 2: Based on the cosine similarity between the text features and the features of each image block, construct a similarity map representing fine-grained text-image consistency; Step 3: Based on the similarity map, use the configured threshold to filter the image patch features with the highest similarity to the text features, and obtain the sequence index of the image patch; Step 4: Fuse the global features of the image with the image patch features corresponding to the sequence index to obtain multi-granularity fused image features; Step 5: Based on the multi-granularity fused image features, the CLIP model is adjusted and frozen by using a fully connected layer as the sentiment prediction head and the cross-entropy loss function as the sentiment classification loss. Step 6: Obtain the emoji to be recognized and pass it through the currently frozen CLIP model image encoder. Repeat steps 1 to 4 to obtain the fused image features to be recognized. Step 7: Embed the features of the image to be identified into a preset prompt template to form an input structure; Step 8: Input the input structure into the LLM model for fine-tuning and predict the sentiment category; Step 9: Adjust the LLM model and iterate it based on the emotion category prediction results until the LLM model meets the preset requirements in emotion recognition.

2. The emoji emotion recognition method based on large-model multimodal multi-granularity feature fusion according to claim 1, characterized in that, The method further includes: For emoji images in the acquired emoji dataset that do not have accompanying text annotations, use the pre-trained BLIP-2 model to generate descriptive text for these images to augment the image-text data pairs in the dataset.

3. The emoji emotion recognition method based on large-model multimodal multi-granularity feature fusion according to claim 2, characterized in that, In step 1, The image and text data are output as text features F by multiple ViT modules of the encoder. text Image features F img The hidden layer image features are represented as follows: in, This represents the global feature vector, and L represents the number of ViT modules; Image patch features are:

4. The emoji emotion recognition method based on large-model multimodal multi-granularity feature fusion according to claim 3, characterized in that, Step 2 includes: Where Sim is the fine-grained similarity graph, · represents the vector inner product operation, and ||·||2 represents the Euclidean norm of the vector.

5. The emoji emotion recognition method based on large-model multimodal multi-granularity feature fusion according to claim 4, characterized in that, In step 4, the multi-granularity fused image features are determined by the following formula: in, represents the global feature vector output by layer L-1, and k represents the k image patch vectors with the highest cosine similarity selected.

6. The emoji emotion recognition method based on large-model multimodal multi-granularity feature fusion according to claim 5, characterized in that, Step 5 includes: The probability distribution for each emotion category is determined as follows: Where W is the weight matrix of the linear layer, C represents the total number of sentiment categories, and P i This represents the probability that the model predicts the i-th sentiment category; The difference between the model output probability distribution P and the true label distribution Y is calculated using the cross-entropy loss function, and the CLIP is fine-tuned by minimizing the sentiment classification loss. The cross-entropy loss function L is shown in the following equation: The CLIP model is adjusted by minimizing the cross-entropy loss function to map text-image pair features to the correct sentiment category. Freeze the adjusted CLIP model.

7. The emoji emotion recognition method based on large-model multimodal multi-granularity feature fusion according to claim 6, characterized in that, In step 7, the preset prompt template includes: the fused image features to be identified, the corresponding text, and a specific sentiment prediction prompt.

8. The emoji emotion recognition method based on large-model multimodal multi-granularity feature fusion according to claim 7, characterized in that, In step 9, based on the predicted sentiment category, the loss function weights and hyperparameters of the LLM model are adjusted iteratively to improve the model until it achieves optimal performance on the sentiment recognition task.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the emoji emotion recognition method based on a large model as described in any one of claims 1 to 8.

10. A computer storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the emoji emotion recognition method based on a large model as described in any one of claims 1 to 8.

Images