Sentiment analysis method based on prototype guide mode fusion and prompt enhancement

Abstract

The invention discloses a sentiment analysis method based on prototype guide mode fusion and prompt enhancement, and constructs a multi-mode sentiment analysis network which comprises a multi-mode coding module, a prototype guide mode fusion module, a dynamic mode weight adjustment mechanism and a context prompt generation module. The method comprises the following steps: firstly, extracting semantic features of each mode by using a multi-mode encoder, and constructing a prototype feature library based on a labeled sample to describe typical representations of different modes under each category; and then, dynamically evaluating modal contribution through prototype similarity to realize modal adaptive fusion. Furthermore, a context prompt is generated according to a similarity retrieval result of the input sample and the prototype library, and the pre-training language model is guided to complete sentiment classification. According to the method, the problems of modal inconsistency, information redundancy, weak small sample generalization and the like can be effectively relieved, and the accuracy and robustness of sentiment analysis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and multimodal sentiment computing, and in particular to a sentiment analysis method based on prototype-guided modal fusion and prompt enhancement. Background Art

[0002] Multimodal sentiment analysis is a key technology for improving the emotional understanding capabilities of intelligent human-computer interaction systems. The user's emotional state directly impacts the quality of system responses and the interactive experience. Traditional methods rely on a single modality, using sentiment dictionaries or lexical features for recognition. However, emotions inherently have multidimensional expression properties, making it difficult for a single modality to accurately capture complex semantics. This is particularly true in scenarios with ambiguous context or overlapping emotions. To address this, research is increasingly shifting towards multimodal modeling, integrating data from multiple sources, such as text and images, to improve recognition accuracy.

[0003] Existing multimodal sentiment analysis methods mainly fall into two categories: static fusion models and dynamic interaction models. The former typically uses feature splicing or attention mechanisms for unified classification, while the latter achieves semantic complementarity and feature reconstruction through cross-attention or modal alignment, enhancing fusion expression capabilities. Although relevant methods have made certain progress, the following challenges still exist: First, models generally rely on a large number of labeled samples, while actual sentiment data often suffers from problems such as uneven categories, missing modalities, and high annotation costs, resulting in overfitting and poor generalization under small sample conditions; second, different modalities have differences and uncertainties in sentiment expression. Traditional fusion methods use fixed structures or static weights, ignoring the imbalance of modal contributions and being easily disturbed by the dominant modality, which affects classification results; third, most methods lack contextual semantic guidance mechanisms and cannot effectively utilize supporting samples or historical semantic information for auxiliary modeling. They are unstable when dealing with complex contexts and polysemous emotional expressions, and the prediction results are poorly interpretable. Summary of the Invention

[0004] The purpose of the present invention is to overcome the problems of insufficient modal fusion, information imbalance and poor generalization ability under small sample conditions in existing multimodal sentiment analysis methods. A sentiment analysis method based on prototype-guided modal fusion and prompt enhancement is proposed. This method realizes modal semantic alignment by constructing category prototypes, adjusts modal contributions by adopting a dynamic modal weight adjustment mechanism, and introduces contextual prompt templates to guide language model classification, effectively improving the accuracy and robustness of sentiment recognition and enhancing the adaptability of the model in complex contexts and sample-scarce scenarios.

[0005] To achieve the above-mentioned purpose, the technical solution provided by the present invention is: a sentiment analysis method based on prototype-guided modal fusion and prompt enhancement, which constructs a multimodal sentiment analysis network, which includes a multimodal encoding module, a prototype-guided modal fusion module, a dynamic modal weight adjustment mechanism and a context prompt generation module; the multimodal encoding module is used to respectively receive text modal data and image modal data in the input sample, and extract high-level semantic features respectively through a preset text encoder and image encoder, wherein the text encoder outputs a context-aware semantic vector, and the image encoder outputs a spatially aware visual representation; the prototype-guided modal fusion module is used to receive the high-level semantic features of text and images extracted by the multimodal encoding module, and construct a prototype feature library of the corresponding category based on the emotion category information marked in the training set, and the prototype feature library is used to extract high-level semantic features of text and images extracted by the multimodal encoding module. The prototype feature representation in the feature library is the central semantic representation of the modal features under a specific category, which is used to capture the typical structure of each category in different modalities. By calculating the similarity between the modal features of the input sample and the prototype features of each category, the matching degree between the sample and different categories is measured, thereby achieving effective alignment and fusion of text and image modalities in the semantic space; the dynamic modal weight adjustment mechanism is used to adaptively calculate the fusion weight of the text modality and the image modality based on the semantic strength of different modalities of the input sample and the distribution of the modal similarity to the prototypes of each category; the contextual prompt generation module is used to retrieve similar samples with similar semantic features to the current input sample from the prototype feature library, and construct a structured contextual prompt template based on its modal features and category labels. The contextual prompt template integrates the text content, image description and emotional label of the semantically close sample;

[0006] The specific implementation of the sentiment analysis method includes the following steps:

[0007] S1: Receives and processes multimodal input samples containing multimodal information, including textual and image modal information, and extracts high-level semantic features of the textual and image modalities respectively through the multimodal encoding module;

[0008] S2: Use the high-level semantic features of the text modality and image modality obtained in step S1 to build a prototype feature library of the corresponding category, and use the prototype-guided modal fusion module and the dynamic modal weight adjustment mechanism to fuse the multimodal information, guide the multimodal sentiment analysis network to learn semantic consistency and modal balance expression, and optimize the various parameters of the multimodal sentiment analysis network to complete the training;

[0009] S3: Apply the multimodal sentiment analysis network trained in step S2 to complete the modal feature extraction and fusion representation construction of the input sample, retrieve similar samples in the prototype feature library based on the fusion representation, and use the context prompt generation module to construct a context prompt template. The context prompt template is input into the pre-trained language model for sentiment classification and the prediction result is output.

[0010] Furthermore, in step S1, the multimodal input sample is text-image pair data containing emotion category labels, and the emotion category labels are multi-category emotion labels; the number of sample selections K of a single emotion category is pre-set, and K text-image pair samples are selected from each category of emotion labels, and the selected samples are called support samples, which are merged to form a support set, so that the support set contains a total of 3×K support samples; the remaining samples that are not selected are called query samples, which constitute a query set, wherein the support set is used to generate a prototype feature library and construct a context prompt template, and the query set is used to calculate the outer layer loss in the training phase and update the parameters of the multimodal emotion analysis network; the support set and the query set are input to the multimodal encoding module to extract high-level semantic features of the text modality and the image modality, respectively, and are denoted as f text With f image , providing basic feature representation for subsequent prototype construction and fusion; the text encoder is a language model based on the Transformer structure, and the image encoder is an image model based on the visual Transformer structure. The extracted features are processed by L2 normalization to maintain the consistency of feature scale between modalities.

[0011] Furthermore, the specific steps of step S2 are as follows:

[0012] S21: In the prototype-guided modality fusion module, for each emotion category, the text modality and image modality features of all supporting samples in the category are averaged and aggregated to generate a category prototype feature library, which is used to characterize the typical representation of different emotion categories in each modality. The prototype vectors of the text modality and image modality of each input sample are They are defined as:

[0013]

[0014] Where D c represents the c-th class sample set, N c is the number of class samples, and Represent the text modality and image modality feature representation of sample i respectively;

[0015] S22: For each support sample, calculate the similarity between the image modality feature, text modality feature and the corresponding category prototype feature, and the text prototype similarity Image prototype similarity They are defined as:

[0016]

[0017] Where exp(·) represents the exponential function, Represents the Euclidean square distance. The prototype vector under each mode is calculated based on the Euclidean square distance to calculate the similarity score with the input sample. The higher the similarity score, the more likely the sample belongs to the category.

[0018] S23: Based on the above text prototype similarity and image prototype similarity, in order to enhance the alignment ability between modalities and improve the training stability, the modality-guided prototype loss function is introduced. This loss function maximizes the similarity between the input sample and its true category prototype and minimizes the similarity with other category prototypes to achieve supervision constraints on modal semantic consistency; the prototype loss is calculated separately for the text modality and the image modality to reflect the degree of deviation of their respective features from the semantic prototype; the text prototype loss of the text modality and the image modality is Image prototype loss The calculations are as follows:

[0019]

[0020] Where log(·) represents the natural logarithm function, y i represents the true category label of sample i, t represents the total number of emotion categories, represents the sum of all categories, N is the total number of samples in the support set; the above prototype loss encourages the feature vector of each modality to be as close as possible to the prototype of its category;

[0021] S24: A modal weight calculation strategy based on feature amplitude is introduced into the dynamic modal weight adjustment mechanism. Specifically, the fusion weights of text and image modalities are adaptively allocated based on the L2 norm of each modality, i.e., the feature strength. When the modal weight difference exceeds a threshold, a balancing factor is introduced to suppress the dominant modal weight and enhance the weak modal weight, thereby achieving dynamic adjustment of modal contribution and improving the robustness of fusion expression. The modal weight calculation is as follows:

[0022]

[0023] In the formula, ||·||2 represents the L2 norm, which is used to measure the amplitude of the modal characteristics, ω text and ω image are the fusion weights of text modality and image modality respectively; if ω text >ω image , then ω text is the dominant mode, otherwise ω imageis the dominant mode; if the modal balance factor Δω=|ω text -ω image | exceeds the preset threshold θ′, the adjustment coefficient γ=1+Δω is introduced to suppress the weight of the dominant mode, thereby achieving weight balance and finally fusion feature f fusion Expressed as:

[0024] f fusion =(ω text / γ)·f text +(ω image / γ)·f image

[0025] S25: Construct a joint loss function for the training phase, use the standard cross entropy loss function for label prediction supervision, and jointly optimize the text prototype loss, image prototype loss and modality balance factor to construct a total loss function Defined as:

[0026] Where, is the standard cross entropy loss, which is used to optimize the final classification output; α, β, and γ are adjustable hyperparameter weight coefficients; Δω is the modality balance factor, which is used to penalize the unbalanced contribution between modalities;

[0027] S26: Training phase, the above total loss function It is used to guide the parameter update of the multimodal sentiment analysis network and improve the comprehensive ability of the multimodal sentiment analysis network in semantic consistency modeling, modal fusion quality and classification accuracy. In order to optimize the above total loss function, the training process adopts a gradient descent optimization strategy based on back propagation. All learnable parameters in the multimodal sentiment analysis network are denoted as θ. In each training iteration, according to the total loss function Find the gradient of the parameter θ and obtain the direction vector Based on the preset learning rate η, the parameters are optimized according to the following update rules:

[0028]

[0029] In the formula, η is used to control the step size of each round of parameter update; the parameters of the multimodal sentiment analysis network are continuously optimized through the above iterative process, and gradually minimized Finally, the training of the multimodal sentiment analysis network is completed.

[0030] Furthermore, the specific steps of step S3 are as follows:

[0031] S31: Receive input sample x of text-image to be predicted q , input them into the text encoder and image encoder respectively, extract the corresponding semantic features, recorded as and in Represents the text semantic vector obtained by the text encoder after the input text, Represents the image semantic vector obtained by the image encoder after the image modality is passed;

[0032] S32: Utilize the prototype feature library built during the training phase Where M represents the total number of emotion categories. The similarity between the above input sample and each category prototype in each modality is calculated respectively to obtain the matching score of the sample to be predicted under each emotion category. The similarity adopts the exponential function form of negative Euclidean distance. The text semantic vector of the input sample is Prototype vector with text Similarity score And the image semantic vector of the input sample With the image prototype vector Similarity score Calculated as follows:

[0033]

[0034] S33: Perform weighted averaging of the similarities of the two modalities based on the modal fusion weights obtained in the training phase to generate a fusion similarity score. Based on the modal similarity calculation results in step S32 and the modal fusion weights ω determined in the training phase, text With ω image , calculate the weighted fusion similarity sim of each category c :

[0035]

[0036] Where, ω t and ω i are the dynamic fusion weights of text modality and image modality, satisfying ω t +ω i =1;

[0037] S34: Based on the fusion similarity, retrieve the h support samples that are most similar to the input sample for each category c from the training samples to form the prompt context set of the query sample Denoted as:

[0038]

[0039] Where, t j represents the text input of the jth support sample, v j The image input content of the j-th support sample is converted into a natural language description through the image caption generation model, y j represents the true sentiment label of the j-th support sample;

[0040] S35: Gather Support sample splicing in building context hint template The contextual prompt template is organized as a structured language input sequence, which is input into the pre-trained language model Among them This means that the support samples are converted into the language model input format according to a fixed template, and the image modality is converted into a language description to enhance the language model's ability to understand the context;

[0041] S36: Execute the inference process. The language model uses mask filling or autoregressive generation to classify the emotional category of the input sample according to the semantics and label relationship in the prompt to obtain the predicted probability distribution of the emotional category.

[0042]

[0043] Select the final output prediction category That is, the sentiment category with the highest probability completes the final prediction.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] 1. The prototype-guided modal fusion module designed in this invention realizes semantic alignment and discriminative feature extraction between different modalities by constructing category prototype features, alleviating the problem of inconsistent modal expressions and effectively improving the semantic consistency of fusion features.

[0046] 2. The dynamic modal weight adjustment mechanism proposed in this invention can adaptively allocate modal fusion weights according to the modal feature amplitude and semantic similarity, dynamically adjust the contribution of each modality in the final representation, avoid interference from the dominant modality, and enhance the stability and robustness of the model.

[0047] 3. The contextual prompt generation module designed in the present invention can retrieve supporting information related to the current sample from the prototype feature library, construct a structured prompt template and guide the pre-trained language model to perform sentiment classification, significantly improving the model's understanding ability and prediction accuracy in complex contexts.

[0048] 4. The present invention has good adaptability to small samples. Through the joint modeling of prototype construction and prompt guidance mechanism, it can still maintain high recognition performance in data-scarce scenarios and has strong versatility and practical value.

[0049] In summary, this paper solves the problems of existing methods such as modal imbalance, semantic bias and performance degradation under small sample conditions through innovative designs such as prototype-guided fusion, dynamic weight adjustment and prompt enhancement, and significantly improves the accuracy, robustness and generalization ability of multimodal sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a framework diagram of the method of the present invention.

[0051] Figure 2 Schematic diagram of the training phase of the multimodal sentiment analysis network.

[0052] Figure 3 Schematic diagram of the inference stage of multimodal sentiment analysis network. DETAILED DESCRIPTION

[0053] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0054] like Figure 1As shown, this embodiment discloses a sentiment analysis method based on prototype-guided modal fusion and prompt enhancement, and constructs a multimodal sentiment analysis network, which includes a multimodal encoding module, a prototype-guided modal fusion module, a dynamic modal weight adjustment mechanism, and a contextual prompt generation module, which is used to effectively improve the accuracy and generalization ability of sentiment recognition in scenarios where the number of samples is limited and the modal expression capabilities differ significantly. The multimodal encoding module is used to respectively receive text modal data and image modal data in the input sample, and respectively extract high-level semantic features through a preset text encoder and image encoder, wherein the text encoder outputs a context-aware semantic vector, and the image encoder outputs a spatially aware visual representation. The extracted features are normalized and then passed to subsequent modules for fusion and matching. The prototype-guided modal fusion module is used to receive the high-level semantic features of text and images extracted by the multimodal encoding module, and construct a prototype feature library of the corresponding category based on the emotional category information annotated in the training set. The prototype feature representation in the prototype feature library is the central semantic representation of the modal features under a specific category, which is used to capture the typical structure of each category in different modalities. By calculating the similarity between the input sample modal features and the prototypes of each category, the degree of matching between the sample and the different categories is measured, thereby achieving effective alignment and fusion of text and image modalities in the semantic space. The dynamic modal weight adjustment mechanism is used to adaptively calculate the fusion weight of the text modality and the image modality based on the semantic strength of different modalities of the input sample (such as the modal feature amplitude) and the similarity distribution of the modality to the prototypes of each category. When a certain modality contributes strongly to emotional expression but there is expression deviation or interference, the mechanism automatically reduces its dominance and enhances the influence of the weak modality in the fusion representation to alleviate problems such as emotional expression deviation and modality dominance, and enhance the robustness and discriminability of the fusion feature. The contextual prompt generation module is used to retrieve similar samples with similar semantic features to the current input sample from the prototype feature library, and construct a structured contextual prompt template based on their modal features and category labels. The contextual prompt template integrates the text content, image description and emotional label of the semantically similar samples, and passes it into the pre-trained language model as input to guide the model to understand the contextual emotional structure and improve classification accuracy.

[0055] The specific implementation of the sentiment analysis method includes the following steps:

[0056] S1: Receives and processes multimodal input samples, and extracts high-level semantic features of text modality and image modality respectively through the multimodal encoding module;

[0057] The multimodal input samples are text-image pair data containing emotion category labels, and the emotion category labels are multi-category emotion labels; the number of sample selections K of a single emotion category is pre-set, and K text-image pair samples are selected from each category of emotion labels, and the selected samples are called support samples, which are merged to form a support set, so that the support set contains a total of 3×K support samples; the remaining unselected samples are called query samples, which constitute a query set, wherein the support set is used to generate a prototype feature library and construct a context prompt template, and the query set is used to calculate the outer layer loss in the training phase and update the parameters of the multimodal emotion analysis network; the support set and the query set are input to the multimodal encoding module to extract high-level semantic features of the text modality and the image modality, respectively, and are denoted as f text With f image , providing basic feature representation for subsequent prototype construction and fusion; the text encoder is a language model based on the Transformer structure, and the image encoder is an image model based on the visual Transformer structure. The extracted features are processed by L2 normalization to maintain the consistency of feature scale between modalities.

[0058] S2: Training phase of the multimodal sentiment analysis network:

[0059] See Figure 2 As shown in the figure, a category prototype feature library is constructed using multimodal high-level semantic features, and a prototype-guided modal fusion module and a dynamic modal weight adjustment mechanism are used to fuse multimodal information, guide the multimodal sentiment analysis network to learn semantic consistency and modal balance expression, and optimize the various parameters of the multimodal sentiment analysis network to complete training. The training includes the following steps:

[0060] S21: In the prototype-guided modality fusion module, for each emotion category, the text modality and image modality features of all supporting samples in the category are averaged and aggregated to generate a category prototype feature library, which is used to characterize the typical representation of different emotion categories in each modality. The prototype vectors of the text modality and image modality of each input sample are They are defined as:

[0061]

[0062] Where D c represents the c-th class sample set, N c is the number of class samples, and Represent the text modality and image modality feature representation of sample i respectively;

[0063] S22: For each support sample, calculate the similarity between the image modality feature, text modality feature and the corresponding category prototype feature, and the text prototype similarity Image prototype similarity They are defined as:

[0064]

[0065] Where exp(·) represents the exponential function, Represents the Euclidean square distance. The prototype vector under each mode is calculated based on the Euclidean square distance to calculate the similarity score with the input sample. The higher the similarity score, the more likely the sample belongs to the category.

[0066] S23: Based on the above text prototype similarity and image prototype similarity, in order to enhance the alignment ability between modalities and improve the training stability, the modality-guided prototype loss function is introduced. This loss function maximizes the similarity between the input sample and its true category prototype and minimizes the similarity with other category prototypes to achieve supervision constraints on modal semantic consistency; the prototype loss is calculated separately for the text modality and the image modality to reflect the degree of deviation of their respective features from the semantic prototype; the text prototype loss of the text modality and the image modality is Image prototype loss The calculations are as follows:

[0067]

[0068] Where log(·) represents the natural logarithm function, y i represents the true category label of sample i, t represents the total number of emotion categories, represents the sum of all categories, N is the total number of samples in the support set; the above prototype loss encourages the feature vector of each modality to be as close as possible to the prototype of its category;

[0069] S24: To address the problem of modal information imbalance in multimodal sentiment analysis and achieve dynamic balance of information between modalities, a modal weight calculation strategy based on feature amplitude is introduced into the dynamic modal weight adjustment mechanism. Specifically, according to the L2 norm of each modality, that is, the feature strength, the fusion weights of text and image modalities are adaptively allocated. When the modal weight difference exceeds the threshold, a balancing factor is introduced to suppress the dominant modal weight and enhance the weak modal weight, thereby achieving dynamic adjustment of modal contribution and improving the robustness of fusion expression. The modal weight calculation is as follows:

[0070]

[0071] In the formula, ||·||2 represents the L2 norm, which is used to measure the amplitude of the modal characteristics, ω text and ω image are the fusion weights of text modality and image modality respectively; if ω text >ω image , then ω textis the dominant mode, otherwise ω image is the dominant mode; if the modal balance factor Δω=|ω text -ω image | exceeds the preset threshold θ′, the adjustment coefficient γ=1+Δω is introduced to suppress the weight of the dominant mode, thereby achieving weight balance and finally fusion feature f fusion Expressed as:

[0072] f fusion =(ω text / γ)·f text +(ω image / γ)·f image

[0073] S25: Construct a joint loss function for the training phase, use the standard cross entropy loss function for label prediction supervision, and jointly optimize the text prototype loss, image prototype loss and modality balance factor to construct the total loss function Defined as:

[0074]

[0075] Where, is the standard cross entropy loss, which is used to optimize the final classification output; α, β, and γ are adjustable hyperparameter weight coefficients; Δω is the modality balance factor, which is used to penalize the unbalanced contribution between modalities;

[0076] S26: Training phase, the above total loss function It is used to guide the parameter update of the multimodal sentiment analysis network and improve the comprehensive ability of the multimodal sentiment analysis network in semantic consistency modeling, modal fusion quality and classification accuracy. In order to optimize the above total loss function, the training process adopts a gradient descent optimization strategy based on back propagation. All learnable parameters in the multimodal sentiment analysis network are denoted as θ. In each training iteration, according to the total loss function Find the gradient of the parameter θ and obtain the direction vector Based on the preset learning rate η, the parameters are optimized according to the following update rules:

[0077]

[0078] In the formula, η is used to control the step size of each round of parameter update; the parameters of the multimodal sentiment analysis network are continuously optimized through the above iterative process, and gradually minimized Finally, the training of the multimodal sentiment analysis network is completed.

[0079] 3) Reasoning phase of the multimodal sentiment analysis network:

[0080] See Figure 3As shown, in the inference stage, the trained multimodal sentiment analysis network completes the modal feature extraction and fusion representation construction of the input sample, retrieves similar samples in the prototype feature library based on the fusion representation, and constructs a contextual prompt template by the contextual prompt generation module. The contextual prompt template is input into the pre-trained language model for sentiment classification and outputs the prediction result, which includes the following steps:

[0081] S31: Receive input sample x of text-image to be predicted q , input them into the text encoder and image encoder respectively, extract the corresponding semantic features, recorded as and in Represents the semantic vector obtained by the input text through the text encoder, Represents the semantic representation of the image modality obtained by the image encoder;

[0082] S32: Utilize the prototype feature library built during the training phase Where M represents the total number of emotion categories. The similarity between the above input sample and each category prototype in each modality is calculated respectively to obtain the matching score of the sample to be predicted under each emotion category. The similarity adopts the exponential function form of negative Euclidean distance. The text semantic vector of the input sample is Prototype vector with text Similarity score And the image semantic vector of the input sample With the image prototype vector Similarity score Calculated as follows:

[0083]

[0084] S33: Perform weighted averaging of the similarities of the two modalities based on the modal fusion weights obtained in the training phase to generate a fusion similarity score. Based on the modal similarity calculation results in step S32 and the modal fusion weights ω determined in the training phase, text With ω image , calculate the weighted fusion similarity sim of each category c :

[0085]

[0086] Where, ω t and ω i are the dynamic fusion weights of text modality and image modality, satisfying ω t +ω i =1;

[0087] S34: Based on the fusion similarity, retrieve the h support samples that are most similar to the input sample for each category c from the training samples to form the prompt context set of the query sample Denoted as:

[0088]

[0089] Where, t j represents the text input of the jth support sample, v j is the image modality content of the jth support sample, which can be converted into a natural language description through the image caption generation model, y j represents the true emotional label of the jth support sample; the retrieved support sample contains text, image and corresponding label information, which constitute the contextual prompt support set of the input sample. The support set is spliced ​​into a structured language input according to a fixed template to construct a multimodal prompt template, in which the image modality can be converted into a language description to enhance the language model's ability to understand the context;

[0090] S35: Gather Support sample splicing in building context hint template The contextual prompt template is organized as a structured language input sequence, which is input into the pre-trained language model Among them This means that the support samples are converted into the language model input format according to a fixed template, and the image modality is converted into a language description to enhance the language model's ability to understand the context;

[0091] S36: Execute the inference process. The language model uses mask filling or autoregressive generation to classify the emotional category of the input sample according to the semantics and label relationship in the prompt to obtain the predicted probability distribution of the emotional category.

[0092]

[0093] Select the final output prediction category That is, the sentiment category with the highest probability completes the final prediction.

[0094] The embodiments of the present invention are evaluated using six multimodal sentiment analysis public benchmark datasets, including three coarse-grained datasets MVSA-Single, MVSA-Multiple, and TumEmo, and three fine-grained datasets Twitter-15, Twitter-17, and MASAD, covering sentiment recognition tasks in various scenarios and with different category granularity, sample size, and modal characteristics.

[0095] In terms of data partitioning, an 8:1:1 ratio is adopted for the training set, validation set, and test set. In order to adapt to the few-sample learning scenario, the experiment uses 1% of the training and validation samples to construct a small sample data subset to ensure a balanced distribution of each category, and constructs two independent and non-overlapping training / validation subsets to enhance the robustness of the evaluation.

[0096] The method of the present invention is trained based on the AdamW optimizer. During the training phase, the learning rate of the prototype-guided modal fusion module is set to 1e-5, the learning rate of the dynamic modal weight adjustment mechanism is adjusted to 1e-6, the batch size is 64, and 10 to 40 rounds of training cycles are set according to the size of different data sets. All experiments are repeated independently for 3 times, and the arithmetic mean of the test set accuracy is finally used as the performance evaluation indicator to ensure the statistical stability of the experimental conclusions.

[0097] To systematically evaluate the sentiment discrimination capabilities of the trained multimodal sentiment analysis network proposed in this paper, we conducted multiple comparative experiments on six datasets, using performance evaluation metrics including accuracy and F1 score. By comparing the experimental results of various cutting-edge models on the remaining datasets, we can draw the following observations:

[0098] As shown in the experimental results for the coarse-grained datasets in Table 1, the proposed method maintains a leading position in accuracy and F1 index in the coarse-grained sentiment analysis task, reflecting its stronger modeling capabilities in handling complex semantics, polysemous expressions, and contextual dependencies. The average sentiment recognition accuracy on the three datasets is 70.10%, 68.50%, and 66.33%, respectively.

[0099] As shown in Table 2, the experimental results on the fine-grained datasets show that our method achieves robust performance across all evaluation metrics in fine-grained sentiment analysis tasks. Compared to fine-grained datasets with higher semantic complexity, coarse-grained tasks place higher demands on modality consistency and semantic alignment. The average sentiment recognition accuracy on the three datasets is 70.10%, 66.61%, and 91.55%, respectively.

[0100] Table 1 Experimental results of coarse-grained dataset

[0101]

[0102] Table 2 Experimental results on fine-grained datasets

[0103]

[0104]

[0105] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A sentiment analysis method based on prototype-guided modal fusion and prompt enhancement, characterized by: The method constructs a multimodal sentiment analysis network, which includes a multimodal encoding module, a prototype-guided modal fusion module, a dynamic modal weight adjustment mechanism and a contextual prompt generation module; the multimodal encoding module is used to respectively receive text modal data and image modal data in input samples, and extract high-level semantic features respectively through a preset text encoder and image encoder, wherein the text encoder outputs a context-aware semantic vector, and the image encoder outputs a spatially aware visual representation; the prototype-guided modal fusion module is used to receive the high-level semantic features of text and images extracted by the multimodal encoding module, and construct a prototype feature library of the corresponding category based on the emotion category information marked in the training set, wherein the prototype features in the prototype feature library are represented as the median of the modal features under a specific category. The core semantic representation is used to capture the typical structure of each category in different modalities. By calculating the similarity between the modal features of the input sample and the prototype features of each category, the matching degree between the sample and different categories is measured, thereby achieving effective alignment and fusion of text and image modalities in the semantic space; the dynamic modal weight adjustment mechanism is used to adaptively calculate the fusion weight of the text modality and the image modality according to the semantic strength of the different modalities of the input sample and the distribution of the modal similarity to the prototype of each category; the contextual prompt generation module is used to retrieve similar samples with similar semantic features to the current input sample from the prototype feature library, and construct a structured contextual prompt template based on its modal features and category labels. The contextual prompt template integrates the text content, image description and emotional label of the semantically close sample; The specific implementation of the sentiment analysis method includes the following steps: S1: Receives and processes multimodal input samples containing multimodal information, including textual and image modal information, and extracts high-level semantic features of the textual and image modalities respectively through the multimodal encoding module; S2: Use the high-level semantic features of the text modality and image modality obtained in step S1 to build a prototype feature library of the corresponding category, and use the prototype-guided modal fusion module and the dynamic modal weight adjustment mechanism to fuse the multimodal information, guide the multimodal sentiment analysis network to learn semantic consistency and modal balance expression, and optimize the various parameters of the multimodal sentiment analysis network to complete the training; S3: Apply the multimodal sentiment analysis network trained in step S2 to complete the modal feature extraction and fusion representation construction of the input sample, retrieve similar samples in the prototype feature library based on the fusion representation, and use the context prompt generation module to construct a context prompt template. The context prompt template is input into the pre-trained language model for sentiment classification and the prediction result is output.

2. The sentiment analysis method based on prototype-guided modal fusion and prompt enhancement according to claim 1 is characterized in that: In step S1, the multimodal input sample is text-image pair data containing emotion category labels, and the emotion category labels are multi-category emotion labels; the number of sample selections K of a single emotion category is pre-set, and K text-image pair samples are selected from each category of emotion labels, and the selected samples are called support samples, which are merged to form a support set, so that the support set contains a total of 3×K support samples; the remaining samples that are not selected are called query samples, which constitute a query set, wherein the support set is used to generate a prototype feature library and construct a context prompt template, and the query set is used to calculate the outer layer loss in the training phase and update the parameters of the multimodal emotion analysis network; the support set and the query set are input to the multimodal encoding module to extract high-level semantic features of the text modality and the image modality, respectively, and are denoted as f text With f image , providing basic feature representation for subsequent prototype construction and fusion; The text encoder is a language model based on the Transformer structure, and the image encoder is an image model based on the visual Transformer structure. The extracted features are subjected to L2 normalization to maintain feature scale consistency between modalities.

3. The sentiment analysis method based on prototype-guided modal fusion and prompt enhancement according to claim 2 is characterized in that: The specific steps of step S2 are as follows: S21: In the prototype-guided modality fusion module, for each emotion category, the text modality and image modality features of all supporting samples in the category are averaged and aggregated to generate a category prototype feature library, which is used to characterize the typical representation of different emotion categories in each modality. The prototype vectors of the text modality and image modality of each input sample are They are defined as: Where D c represents the c-th class sample set, N c is the number of class samples, f i text With f i image Represent the text modality and image modality feature representation of sample i respectively; S22: For each support sample, calculate the similarity between the image modality feature, text modality feature and the corresponding category prototype feature, and the text prototype similarity Image prototype similarity They are defined as: Where exp(·) represents the exponential function, Represents the Euclidean square distance. The prototype vector under each mode is calculated based on the Euclidean square distance to calculate the similarity score with the input sample. The higher the similarity score, the more likely the sample belongs to the category. S23: Based on the above text prototype similarity and image prototype similarity, in order to enhance the alignment ability between modalities and improve the training stability, the modality-guided prototype loss function is introduced. This loss function maximizes the similarity between the input sample and its true category prototype and minimizes the similarity with other category prototypes to achieve supervision constraints on modal semantic consistency; the prototype loss is calculated separately for the text modality and the image modality to reflect the degree of deviation of their respective features from the semantic prototype; the text prototype loss of the text modality and the image modality is Image prototype loss The calculations are as follows: Where log(·) represents the natural logarithm function, y i represents the true category label of sample i, t represents the total number of emotion categories, represents the sum of all categories, N is the total number of samples in the support set; the above prototype loss encourages the feature vector of each modality to be as close as possible to the prototype of its category; S24: A modal weight calculation strategy based on feature amplitude is introduced into the dynamic modal weight adjustment mechanism. Specifically, the fusion weights of text and image modalities are adaptively allocated based on the L2 norm of each modality, i.e., the feature strength. When the modal weight difference exceeds a threshold, a balancing factor is introduced to suppress the dominant modal weight and enhance the weak modal weight, thereby achieving dynamic adjustment of modal contribution and improving the robustness of fusion expression. The modal weight calculation is as follows: In the formula, ||·||2 represents the L2 norm, which is used to measure the amplitude of the modal characteristics, ω text and ω image are the fusion weights of text modality and image modality respectively; if ω text >ω image , then ω text is the dominant mode, otherwise ω image is the dominant mode; if the modal balance factor Δω=|ω text -ω image | exceeds the preset threshold θ′, the adjustment coefficient γ=1+Δω is introduced to suppress the weight of the dominant mode, thereby achieving weight balance and finally fusion feature f fusion Expressed as: f fusion =(ω text / c)·f text +(ω image / c)·f image S25: Construct a joint loss function for the training phase, use the standard cross entropy loss function for label prediction supervision, and jointly optimize the text prototype loss, image prototype loss and modality balance factor to construct a total loss function Defined as: Where, is the standard cross entropy loss, which is used to optimize the final classification output. α, β, and γ are adjustable hyperparameter weight coefficients. Δω is the modality balance factor, which is used to penalize the unbalanced contribution between modalities. S26: Training phase, the above total loss function It is used to guide the parameter update of the multimodal sentiment analysis network and improve the comprehensive ability of the multimodal sentiment analysis network in semantic consistency modeling, modal fusion quality and classification accuracy. In order to optimize the above total loss function, the training process adopts a gradient descent optimization strategy based on back propagation. All learnable parameters in the multimodal sentiment analysis network are denoted as θ. In each training iteration, according to the total loss function Find the gradient of the parameter θ and obtain the direction vector Based on the preset learning rate η, the parameters are optimized according to the following update rules: In the formula, η is used to control the step size of each round of parameter update; the parameters of the multimodal sentiment analysis network are continuously optimized through the above iterative process, and gradually minimized Finally, the training of the multimodal sentiment analysis network is completed.

4. The sentiment analysis method based on prototype-guided modal fusion and prompt enhancement according to claim 3 is characterized in that: The specific steps of step S3 are as follows: S31: Receive input sample x of text-image to be predicted q , input them into the text encoder and image encoder respectively, extract the corresponding semantic features, recorded as and in Represents the text semantic vector obtained by the text encoder after the input text, Represents the image semantic vector obtained by the image encoder after the image modality is passed; S32: Utilize the prototype feature library built during the training phase Where M represents the total number of emotion categories. The similarity between the above input sample and each category prototype in each modality is calculated respectively to obtain the matching score of the sample to be predicted under each emotion category. The similarity adopts the exponential function form of negative Euclidean distance. The text semantic vector of the input sample is Prototype vector with text Similarity score And the image semantic vector of the input sample With the image prototype vector Similarity score Calculated as follows: S33: Perform weighted averaging of the similarities of the two modalities based on the modal fusion weights obtained in the training phase to generate a fusion similarity score. Based on the modal similarity calculation results in step S32 and the modal fusion weights ω determined in the training phase, text With ω image , calculate the weighted fusion similarity sim of each category c : Where, ω t and ω i are the dynamic fusion weights of text modality and image modality, satisfying ω t +ω i =1; S34: Based on the fusion similarity, retrieve the h support samples that are most similar to the input sample for each category c from the training samples to form the prompt context set of the query sample Denoted as: Where, t j represents the text input of the jth support sample, v j The image input content of the j-th support sample is converted into a natural language description through the image caption generation model, y j represents the true sentiment label of the j-th support sample; S35: Gather Support sample splicing in building context hint template The contextual prompt template is organized as a structured language input sequence, which is input into the pre-trained language model Among them This means that the support samples are converted into the language model input format according to a fixed template, and the image modality is converted into a language description to enhance the language model's ability to understand the context; S36: Execute the inference process. The language model uses mask filling or autoregressive generation to classify the emotional category of the input sample according to the semantics and label relationship in the prompt to obtain the predicted probability distribution of the emotional category. Select the final output prediction category That is, the sentiment category with the highest probability completes the final prediction.

Images