Robust enhancement method and classification method of multi-modal generative sentiment analysis model

Abstract

The invention provides a robustness enhancement method and a classification method for a multi-modal generative sentiment analysis model, and belongs to the technical field of natural language processing, the method comprises the following steps: obtaining an original input set, and constructing a new input set based on the original input set and a target metamorphic relationship; respectively inputting the original input set and the new input set into an initial multi-modal generative sentiment analysis model to obtain sentiment tags; selecting expected data from the original input set and the new input set based on the emotion labels, and constructing a training set; and training the multi-modal generative emotion analysis model based on the training set and the emotion label corresponding to the training set to obtain a new multi-modal generative emotion analysis model. According to the method, the target metamorphic relation based on attributes is designed, the robustness of the model is effectively analyzed by applying the metamorphic test, and the technical problem that the robustness of the model cannot be effectively analyzed in an existing multi-modal target-level sentiment classification method is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a robust enhancement method and a classification method for a multimodal generative sentiment analysis model. Background Art

[0002] Social media is a core platform for users to share their experiences and opinions, providing rich multimodal data, including text, images and videos, which provides new opportunities for sentiment analysis. The appeal of multimodal content lies in its intuitiveness. For example, users will browse product reviews before purchasing a product. Images and videos can show product features more clearly and have a significant impact on consumer decisions.

[0003] Multimodal sentiment analysis comprehensively analyzes data from different modalities to fully understand user emotions, and has important application value in brand monitoring, public opinion analysis, etc. It helps grasp consumer needs and preferences and optimize product and market strategies. Multimodal sentiment analysis includes subtasks such as sentiment classification, irony recognition, and time series analysis. Among them, sentiment classification is the basis, while multimodal target-level sentiment classification (TMSC) is a fine-grained task of sentiment classification, focusing on the sentiment judgment of specific targets based on multimodal content.

[0004] Existing TMSC research is carried out in a decentralized manner from two tasks, aspect level and entity level, according to the target type. This is because the aspect itself has semantics and can be used as a separate text input, while the core of the entity type is how to better understand its semantic information with the help of the context of the entity. TMSC research has made significant progress, from long short-term memory networks to BERT-based and BART-based pre-training models, demonstrating the development of deep learning and multimodal technology.

[0005] The free expression characteristics and weak logical constraints of Internet language put forward higher requirements on the robustness of sentiment analysis models. Unlike traditional written language, the flexibility and randomness of Internet language require the model to adapt to the ever-changing language usage habits.

[0006] Traditional methods for evaluating the robustness of NLP (Natural Language Processing) models include adversarial attacks and data distribution shift. However, adversarial attacks require artificially designed adversarial perturbations, and data distribution shift requires the collection of a large amount of data to simulate changes in data distribution.

[0007] In the field of multimodal target-level sentiment classification, researchers are committed to exploring how to effectively integrate text and image features to achieve accurate prediction of sentiment. Although existing technologies have made progress, they still face challenges such as how to integrate and process different target types and how to improve the model's robustness to network language.

[0008] In summary, the existing multimodal target-level sentiment classification methods have the problem of being unable to effectively analyze the robustness of the model. Summary of the invention

[0009] In view of this, it is necessary to provide a robust enhancement method and classification method for a multimodal generative sentiment analysis model to solve the technical problem that the existing multimodal target-level sentiment classification method cannot effectively analyze the robustness of the model.

[0010] In order to solve the above problems, on the one hand, the present invention provides a robust enhancement method for a multimodal generative sentiment analysis model, comprising: Acquire an original input set, and construct a new input set based on the original input set and a target metamorphosis relationship; the original input set includes text and images; Inputting the original input set and the new input set into an initial multimodal generative sentiment analysis model respectively to obtain sentiment labels; Based on the emotion label, data meeting expectations are selected from the original input set and the new input set to construct a training set; The multimodal generative sentiment analysis model is trained based on the training set and the sentiment labels corresponding to the training set to obtain a new multimodal generative sentiment analysis model.

[0011] In a possible implementation, the target transformation relationship includes: The consistent target transformation relationship under sentence order changes, the consistent target transformation relationship under key sentence relocation, the consistent target transformation relationship under entity replacement, and the inconsistent target transformation relationship under modifier reversal.

[0012] In a possible implementation, the initial multimodal generative sentiment analysis model is a multi-layer attention network; the multimodal generative sentiment analysis model is trained based on the training set and the sentiment label corresponding to the training set to obtain a new multimodal generative sentiment analysis model, including: Inputting the text in the training set into the task paraphrase multilayer perceptron to obtain a new text set, and performing feature extraction on the new text set to obtain text features; Adding continuous prompts to the images in the training set to obtain a new image set, and performing feature extraction on the new image set to obtain image features; A multi-layer attention network is trained based on the text features, the image features and the corresponding sentiment labels to obtain a new multimodal generative sentiment analysis model.

[0013] In a possible implementation, a multi-layer attention network is trained based on the text features, the image features and the corresponding sentiment labels to obtain a new multimodal generative sentiment analysis model, including: Inputting the text features and the image features into a multi-layer attention network to train the multi-layer attention network, and during the training process, cross-attentionally fusing the text modality features and the image modality features output by the multi-layer attention network to obtain multi-modal fusion features; Based on the multimodal fusion feature, determining the cross entropy loss between the predicted emotion label and the real emotion label corresponding to the text feature and the image feature; Based on the cross entropy loss, the trained multi-layer attention network is optimized to obtain a new multimodal generative sentiment analysis model.

[0014] In a possible implementation, determining the cross entropy loss between the predicted emotion label and the real emotion label corresponding to the text feature and the image feature based on the multimodal fusion feature includes: Inputting the hidden state of the multimodal fusion feature into a multi-layer perceptron to obtain a predicted score probability; Based on the predicted score probability, a cross entropy loss between the predicted emotion label and the true emotion label corresponding to the text feature and the image feature is calculated.

[0015] In a possible implementation, constructing a new input set based on the original input set and the target transformation relationship includes: In the case where the original input set is text, generating seed prompts and keywords of the original input set based on the task description; deriving the most task-relevant paraphrase cue from the seed cue; Based on the paraphrase prompt, the keyword and the original input set, and in combination with the target metamorphosis relationship, a new input set is constructed.

[0016] In a possible implementation, constructing a new input set based on the original input set and the target transformation relationship includes: In the case where the original input set is an image, the original input set is divided into a plurality of regions, and a vector is initialized for each region to obtain a series of continuous vectors; Based on the continuous vector and the target transformation relationship, a new input set is constructed.

[0017] In a possible implementation, constructing a new input set based on the continuous vector and the target transformation relationship includes: An initial vector based on the image embedding in the original input set is used as a prefix of the continuous vector; A new input set is constructed based on the continuous vector and the prefix of the continuous vector and in combination with a target metamorphosis relationship.

[0018] On the other hand, the present invention also provides a multimodal sentiment classification method, comprising: Obtain the data set to be identified; The data set to be identified is input into the multimodal generative sentiment analysis model obtained by the robust enhancement method of the multimodal generative sentiment analysis model described in any one of the above items to obtain the sentiment type.

[0019] On the other hand, the present invention also provides a robust enhancement device for a multimodal generative sentiment analysis model, comprising: A data acquisition module is used to acquire an original input set, and construct a new input set based on the original input set and a target metamorphosis relationship; the original input set includes text and images; A label extraction module, used for inputting the original input set and the new input set into the initial multimodal generative sentiment analysis model respectively to obtain sentiment labels; A training set construction module, used for selecting expected data from the original input set and the new input set based on the emotion label to construct a training set; The model training module is used to train the multimodal generative sentiment analysis model based on the training set and the sentiment tags corresponding to the training set to obtain a new multimodal generative sentiment analysis model.

[0020] The beneficial effects of adopting the above implementation method are as follows: the robust enhancement method and classification method of the multimodal generative sentiment analysis model provided by the present invention, by obtaining the original input set, based on the original input set and the target metamorphosis relationship, construct a new input set; the original input set and the new input set are respectively input into the initial multimodal generative sentiment analysis model to obtain sentiment labels; based on the sentiment labels, the expected data is selected from the original input set and the new input set to construct a training set; based on the training set and the sentiment labels corresponding to the training set, the multimodal generative sentiment analysis model is trained to obtain a new multimodal generative sentiment analysis model. Through metamorphosis testing, the robustness of the model is analyzed, and the model is trained and strengthened in a targeted manner. Aiming at the characteristics of free expression of network language, a target metamorphosis relationship based on attributes is designed. The robustness of the model is effectively analyzed by using metamorphosis testing, and the sentence position, entity and modifier in the sentiment text are comprehensively considered. The model training is enhanced by adding data-level transformation without the need for additional manual annotation. This method not only improves the adaptability and stability of the model to network language, but also provides users with a more accurate sentiment analysis tool, which has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an embodiment of a robust enhancement method for a multimodal generative sentiment analysis model provided by the present invention; Figure 2 A flowchart of another embodiment of the robust enhancement method for the multimodal generative sentiment analysis model provided by the present invention; Figure 3 A flowchart of an embodiment of the multimodal sentiment classification method provided by the present invention; Figure 4 A principle block diagram of an embodiment of a robust enhancement device for a multimodal generative sentiment analysis model provided by the present invention; Figure 5 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more than two.

[0025] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or equipment comprising a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or equipment.

[0026] The naming or numbering of the steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0027] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] The present invention provides a robust enhancement method and a classification method for a multimodal generative sentiment analysis model, which are described below respectively.

[0029] like Figure 1 As shown, the present invention provides a robust enhancement method for a multimodal generative sentiment analysis model, comprising: S101, obtaining an original input set, and constructing a new input set based on the original input set and a target metamorphosis relationship; the original input set includes text and images; S102, inputting the original input set and the new input set into an initial multimodal generative sentiment analysis model respectively to obtain sentiment labels; the multimodal generative sentiment analysis model may also be referred to as an integrated multimodal target-level sentiment classification model; S103, selecting data that meets expectations from the original input set and the new input set based on the emotion label to construct a training set; S104: training the multimodal generative sentiment analysis model based on the training set and the sentiment tags corresponding to the training set to obtain a new multimodal generative sentiment analysis model.

[0030] Wherein, the target transformation relationship includes: The consistent target transformation relationship under sentence order changes, the consistent target transformation relationship under key sentence relocation, the consistent target transformation relationship under entity replacement, and the inconsistent target transformation relationship under modifier reversal.

[0031] It can be understood that the input sets in the present invention are all input data sets, including text and pictures.

[0032] The present invention formulates attribute-based target metamorphosis relations, including three consistency and one inconsistency relations, which comprehensively consider the sentence position, entity and modifier in the sentiment text to evaluate the robustness of the model to the sentiment text. The consistency metamorphosis relation is used to test the stability of the model output, while the inconsistency metamorphosis relation is used to explore the adaptability of the model to output changes.

[0033] Through the framework of target metamorphosis relationship, the present invention designs a robustness analysis process including consistency and inconsistency relationship. Through metamorphosis test, the robustness of the model is analyzed, and the model is trained and strengthened in a targeted manner, which enhances the adaptability and stability of the model to non-standard network languages.

[0034] The present invention aims to solve the problem of sentiment classification of specific targets (aspects / entities) in multimodal content, with a particular focus on the impact of free expression of online language on model robustness. To this end, the present invention proposes a robust target multimodal sentiment classification method, which is specially designed to cope with the situation where the target type has little impact on sentiment classification.

[0035] Existing studies usually regard aspect-level and entity-level sentiment classification as independent tasks, ignoring the close connection between the target sentiment category and its context. The present invention reduces the impact of target type on sentiment classification results through an integrated target classification strategy. The non-standardization of Internet language poses new challenges to the robustness of the model. The present invention designs the consistency and inconsistency target metamorphosis relationship that conforms to the characteristics of Internet language, and uses metamorphosis testing to evaluate and enhance the adaptability of the model to Internet language.

[0036] In some embodiments, the initial multimodal generative sentiment analysis model is a multi-layer attention network; training the multimodal generative sentiment analysis model based on the training set and the sentiment label corresponding to the training set to obtain a new multimodal generative sentiment analysis model includes: Inputting the text in the training set into the task paraphrase multilayer perceptron to obtain a new text set, and performing feature extraction on the new text set to obtain text features; Adding continuous prompts to the images in the training set to obtain a new image set, and performing feature extraction on the new image set to obtain image features; A multi-layer attention network is trained based on the text features, the image features and the corresponding sentiment labels to obtain a new multimodal generative sentiment analysis model.

[0037] It is understandable that the multimodal aspect-level sentiment classification task is to determine the sentiment category of a specific aspect feature based on the multimodal input content. The sentiment category of a specific target is closely related to the contextual information of the target, and is less affected by the target type (aspect or entity). Existing research has limitations in integrating the contextual information of the target and fails to fully capture the emotional nuances of the target.

[0038] In this embodiment, these two target types can be processed simultaneously to more fully understand the multimodal content.

[0039] In some embodiments, reference Figure 2, based on the text features, the image features and the corresponding sentiment labels, a multi-layer attention network is trained to obtain a new multimodal generative sentiment analysis model, including: Inputting the text features and the image features into a multi-layer attention network to train the multi-layer attention network, and during the training process, cross-attentionally fusing the text modality features and the image modality features output by the multi-layer attention network to obtain multi-modal fusion features; Based on the multimodal fusion feature, determining the cross entropy loss between the predicted emotion label and the real emotion label corresponding to the text feature and the image feature; Based on the cross entropy loss, the trained multi-layer attention network is optimized to obtain a new multimodal generative sentiment analysis model.

[0040] In some embodiments, determining the cross entropy loss between the predicted emotion label and the real emotion label corresponding to the text feature and the image feature based on the multimodal fusion feature includes: Inputting the hidden state of the multimodal fusion feature into a multi-layer perceptron to obtain a predicted score probability; Based on the predicted score probability, a cross entropy loss between the predicted emotion label and the true emotion label corresponding to the text feature and the image feature is calculated.

[0041] In some embodiments, constructing a new input set based on the original input set and the target metamorphosis relationship includes: In the case where the original input set is text, generating seed prompts and keywords of the original input set based on the task description; deriving the most task-relevant paraphrase cue from the seed cue; Based on the paraphrase prompt, the keyword and the original input set, and in combination with the target metamorphosis relationship, a new input set is constructed.

[0042] It can be understood that this embodiment involves a target paraphrase prompt template: by converting the task description into a seed prompt and designing a paraphrase rule to generate a variety of semantic prompts, the model's parsing accuracy for the target semantics is enhanced. The paraphrase prompt is used as an input of the text modality and integrated into the pre-trained masked language model (MLM) framework.

[0043] This embodiment constructs a target-specific semantic environment by designing a target paraphrase prompt template, which significantly improves the model's ability to parse the target semantics, thereby improving the accuracy of sentiment classification.

[0044] In some embodiments, constructing a new input set based on the original input set and the target metamorphosis relationship includes: In the case where the original input set is an image, the original input set is divided into a plurality of regions, and a vector is initialized for each region to obtain a series of continuous vectors; Based on the continuous vector and the target transformation relationship, a new input set is constructed.

[0045] In some embodiments, constructing a new input set based on the continuous vector and the target transformation relationship includes: An initial vector based on the image embedding in the original input set is used as a prefix of the continuous vector; A new input set is constructed based on the continuous vector and the prefix of the continuous vector and in combination with a target metamorphosis relationship.

[0046] It can be understood that this embodiment involves visual target prefix optimization: introducing visual prefix prompts to continuously fine-tune multimodal data closely related to the target to improve the quality of feature representation. Based on the pre-trained visual features, new vectors are introduced as prefixes, and only these prefixes are updated during the training process, while the original visual features remain unchanged.

[0047] This embodiment optimizes the visual target prefix, adds continuity prompts to multimodal data, and performs continuity fine-tuning, thereby optimizing the representation of visual features, so that the model can more finely identify multimodal information related to the target.

[0048] In some embodiments, the robust enhancement method of the multimodal generative sentiment analysis model proposed in the present invention is composed of a prompt learning module based on task paraphrase and a target metamorphosis relationship module. The former uses task description to generate paraphrase prompts, combines visual target prefix optimization, and integrates aspects and entity target types. At the same time, the latter designs the target metamorphosis relationship, analyzes the model robustness through metamorphosis test, and improves the model stability.

[0049] The method provided by the present invention has been verified on multiple data sets and has shown good effectiveness and robustness. In terms of accuracy and macro F1 value, the method provided by the present invention can effectively process aspect and entity targets. In addition, the present invention applies the attribute-based target metamorphosis relationship to the robustness analysis of sentiment classification for the first time, verifying the enhancement of model robustness.

[0050] The present invention refers to aspects and entities as targets. Therefore, the multimodal target-level sentiment classification (TMSC) task can be described as: given a multimodal sample set D, for each sample d (d∈D), it contains a text component S, one or more related image components I, and a target T (T, as a subsequence of the text component S, is the core object of sentiment analysis). For T, it has a sentiment tendency identifier Y. The goal is to learn a target-oriented integrated sentiment classifier that can correctly predict the sentiment tendency identifier of T in the input sample X=(S,I,T). The present invention also introduces four target transformation relations to transform the test set to analyze the robustness of the model, and performs training reinforcement under the guidance of the target transformation relations to improve the robustness of the prompt-based multimodal generative sentiment analysis model for online language.

[0051] (1) First, for a given multimodal data X = (S, I, T), the present invention generates a prefix of the text input S through a target paraphrase prompt template. The paraphrase prompt Pi contains a [MASK] flag and is connected to the text S. The text embedding R is obtained through the encoder.

[0052] Specifically, the present invention generates a seed prompt and a keyword K "sentiment" based on the task description. The seed prompt is a concise expression of the core of the task, and subsequently guides the generation of the paraphrase prompt. The final paraphrase rule is as described in Formula 1. The function f represents the relative position of the keyword K, the target T, and T's true tendency emotion Y in the paraphrase prompt, f(Y), f(T), f(K)∈{B,M,E} (representing the beginning, middle, and ending, respectively). The function g(K)={yes,no} is used to indicate whether to replace the keyword K with a synonym. According to the paraphrase rule, a paraphrase prompt closely related to the task can be derived from a seed prompt. Finally, the obtained paraphrase prompt P i As discrete prompts of the text input S, together they form the new input text S new , as shown in Formula 2.

[0053] (Formula 1) (Formula 2) (2) In the visual target prefix prompt, the present invention uses the initial vector V of the pre-trained image embedding E as a continuous prefix. After adding the text embedding R and the image embedding (V+E) to their respective position encodings, the fused vector is obtained through a multi-layer attention network.

[0054] Specifically, for an image, its pre-trained features can be expressed as E pre ={e1 ,e 2 ,…,e r1}, , and |I idx |=r1 represents the length of the pre-trained image features. In order to add continuous hints to the image, the image is divided into multiple regions and a vector is initialized for each region. v i (i∈{1,2,…,r2}), thus forming a series of continuous vectors V soft ={v1,v2,…,v r2}, And use |P idx |=r2 represents the length of the continuous prompt. After obtaining the continuous prompt, the initial vector of the image is embedded V soft As E pre The prefix vector of can be used to obtain the image features that integrate continuous prompts E image , as shown in Formula 3. The representation of image features is shown in Formula 4.

[0055] (Formula 3) (Formula 4) (3) Then obtain the hidden layer vector H and pass it to the MLM head to obtain the predicted score of each word in the vocabulary C at the [MASK] position. Finally, calculate the cross entropy loss LMLM between the predicted result Output and the true sentiment tendency label Y.

[0056] Specifically, during the training process, for the multimodal target-level sentiment classification task, given a four-tuple (S, I, T, Y), the text S is processed by the task paraphrase module to obtain a new text input S new , S new After the text encoder extracts the features, E text After adding continuous prompts to image I, the image features after adding continuous prompts are transformed into E image Transform to and E text In the same input space. Then, a multi-layer attention network is applied to cross-attend fusion of text and image modal features to extract multimodal fusion features. The final hidden state H n After task translation, the multilayer perceptron obtains the predicted score probability, as shown in Formula 5 and Formula 6. Finally, the predicted sentiment label Y is calculated ＇The cross entropy loss between y and the true sentiment label Y is used to optimize the model, as shown in Formula 7. i is the number of true sentiment labels and C is the size of the language model vocabulary.

[0057] (Formula 5) (Formula 6) (Formula 7) (4) After establishing a prompt learning module based on task paraphrase, the present invention introduces four target transformation relations for intensive training, including three consistent target transformation relations and one inconsistent target transformation relation. For a given original input set, a new input set is first constructed based on the target transformation relation. Then the two input sets are respectively input into the prompt-based integrated multimodal target-level sentiment classification module to obtain the original sentiment label and the subsequent sentiment label. Finally, the violation rate is calculated to measure the proportion of test cases that do not meet the expected output after applying the target transformation relation. The target transformation relation includes: (a) Consistency target transformation relation MR1 under sentence order change: Adjusting the order of sentences in the text does not change the sentiment label of the corresponding target.

[0058] The target metamorphosis relationship is based on a key assumption: even if the text review presents a specific inconsistent style, changing the order of the sentences will not affect the judgment of the sentiment tendency of a target. Therefore, this metamorphosis relationship is used to test and strengthen the ability of the prompt-based integrated model in dealing with the consistency of sentiment labels. Given that Internet language does not strictly follow the rules of punctuation, this paper uses the common terminal tag symbols in Internet language, such as "!", ".", and ";" as delimiters to segment and reorder the sentences in the comments.

[0059] (b) Consistent target metamorphosis relation MR2 under key sentence relocation: relocate the key sentences describing the target in the text without changing the sentiment label of the corresponding target.

[0060] In text data, there are often core sentences that directly express satisfaction with a certain target. Even if the relative position of this key sentence is adjusted, the sentiment label will not change the emotional tendency of the target due to changes in the sentences before and after the key sentence. In order to apply this target metamorphosis relationship, the present invention adopts two different methods to identify key sentences that are closely related to specific targets. The first method is to use Word2Vec to encode the entire text and the specific target respectively, and then use WMD to measure the similarity between the target and each sentence to determine the sentence most relevant to the target. The second method is to use BERT to extract the features of the target and the entire text, and identify the key sentences by calculating the cosine similarity between the two. Therefore, the consistent target metamorphosis relationship under key sentence relocation can be divided into: MR21-1, MR21-2, and MR21-3: Use Word2Vec and WMD to identify key sentences and place them at the beginning, middle, and end of the text review, respectively. MR22-1, MR22-2, and MR22-3: Apply BERT and cosine similarity to identify key sentences and place them at the beginning, middle, and end of the text, respectively.

[0061] (c) Consistent target transformation relation MR3 under entity replacement: the entity to be classified is changed without changing the sentiment label.

[0062] Since a single entity itself does not carry any inherent meaning or sentiment, and the sentiment label of an entity usually depends on the evaluation content of the entity, if only the entity to be classified is replaced but the evaluation content of the entity is not changed, the sentiment label of the entity to be classified will not change. The present invention selects other entities of the same type in the same data set for replacement to maintain the consistency of the entity type.

[0063] (d) Inconsistent target transformation relation MR4 under modifier reversal: When the modifier of the target is changed to its antonym, the corresponding target’s sentiment label should become opposite.

[0064] Since the sentiment label is usually determined by the evaluation content of the target, if the descriptive evaluation of the target is reversed, the sentiment label of the target will also become opposite. The present invention searches for subsequences in the text to check whether there are words that match the sentiment dictionary vocabulary. Once a match is found, the match is converted to antonym and the sentiment label of the target is adjusted accordingly.

[0065] In summary, the robust enhancement method of the multimodal generative sentiment analysis model proposed in the present invention realizes the integrated modeling of aspect-level and entity-level sentiment classification tasks under the same language model framework by integrating task paraphrase prompt learning and target transformation relationship modules. At the same time, the robustness of the model is analyzed and optimized by simulating sentence rearrangement, key sentence location, entity replacement and modifier reversal in sentimental texts.

[0066] The present invention effectively realizes the integrated processing of multimodal sentiment classification tasks and improves the flexibility of the model. The designed target metamorphosis relationship module reduces the dependence on manual annotation by adding data-level transformation technology, and improves the efficiency and economy of training. Experimental results show that the method of the present invention exhibits excellent performance on multiple data sets, especially in integrated modeling and sentiment classification. The robustness of the model is also improved after training enhancement.

[0067] like Figure 3 As shown, the present invention also provides a multimodal sentiment classification method, comprising: S301, obtaining a data set to be identified; S302: Input the data set to be identified into a multimodal generative sentiment analysis model obtained by the robust enhancement method of the multimodal generative sentiment analysis model described in any one of the above items to obtain a sentiment type.

[0068] like Figure 4 As shown, the present invention also provides a robust enhancement device 400 for a multimodal generative sentiment analysis model, comprising: The data acquisition module 401 is used to acquire an original input set, and construct a new input set based on the original input set and the target metamorphosis relationship; the original input set includes text and images; A label extraction module 402 is used to input the original input set and the new input set into the initial multimodal generative sentiment analysis model to obtain sentiment labels; A training set construction module 403 is used to select expected data from the original input set and the new input set based on the emotion label to construct a training set; The model training module 404 is used to train the multimodal generative sentiment analysis model based on the training set and the sentiment tags corresponding to the training set to obtain a new multimodal generative sentiment analysis model.

[0069] The robust enhancement device for the multimodal generative sentiment analysis model provided in the above-mentioned embodiment can implement the technical solution described in the embodiment of the robust enhancement method for the multimodal generative sentiment analysis model. The specific implementation principles of the above-mentioned modules or units can refer to the corresponding contents in the embodiment of the robust enhancement method for the multimodal generative sentiment analysis model, which will not be repeated here.

[0070] like Figure 5 As shown, the present invention also provides an electronic device 500. The electronic device 500 includes a processor 501, a memory 502 and a display 503. Figure 5 Only some components of the electronic device 500 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0071] In some embodiments, the memory 502 may be an internal storage unit of the electronic device 500, such as a hard disk or memory of the electronic device 500. In other embodiments, the memory 502 may also be an external storage device of the electronic device 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 500.

[0072] Furthermore, the memory 502 may include both an internal storage unit of the electronic device 500 and an external storage device. The memory 502 is used to store application software installed in the electronic device 500 and various data.

[0073] In some embodiments, the processor 501 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 502, such as the robust enhancement method of the multimodal generative sentiment analysis model in the present invention.

[0074] In some embodiments, the display 503 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 503 is used to display information on the electronic device 500 and to display a visual user interface. The components 501-503 of the electronic device 500 communicate with each other via a system bus.

[0075] In some embodiments of the present invention, when the processor 501 executes the robust enhancement program of the multimodal generative sentiment analysis model in the memory 502, the following steps may be implemented: Acquire an original input set, and construct a new input set based on the original input set and a target metamorphosis relationship; the original input set includes text and images; Inputting the original input set and the new input set into an initial multimodal generative sentiment analysis model respectively to obtain sentiment labels; Based on the emotion label, data meeting expectations are selected from the original input set and the new input set to construct a training set; The multimodal generative sentiment analysis model is trained based on the training set and the sentiment labels corresponding to the training set to obtain a new multimodal generative sentiment analysis model.

[0076] It should be understood that: when the processor 501 executes the robust enhancement program of the multimodal generative sentiment analysis model in the memory 502, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0077] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 500 mentioned, and the electronic device 500 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 500 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0078] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the robust enhancement method of the multimodal generative sentiment analysis model provided by the above methods, the method comprising: Acquire an original input set, and construct a new input set based on the original input set and a target metamorphosis relationship; the original input set includes text and images; Inputting the original input set and the new input set into an initial multimodal generative sentiment analysis model respectively to obtain sentiment labels; Based on the emotion label, data meeting expectations are selected from the original input set and the new input set to construct a training set; The multimodal generative sentiment analysis model is trained based on the training set and the sentiment labels corresponding to the training set to obtain a new multimodal generative sentiment analysis model.

[0079] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0080] The above is a detailed introduction to the robust enhancement method and classification method of the multimodal generative sentiment analysis model provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and scope of application. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A robust enhancement method for a multimodal generative sentiment analysis model, characterized in that: include: Acquire an original input set, and construct a new input set based on the original input set and a target transformation relationship; The original input set includes text and images; Inputting the original input set and the new input set into an initial multimodal generative sentiment analysis model respectively to obtain sentiment labels; Based on the emotion label, data meeting expectations are selected from the original input set and the new input set to construct a training set; The multimodal generative sentiment analysis model is trained based on the training set and the sentiment labels corresponding to the training set to obtain a new multimodal generative sentiment analysis model.

2. The robust enhancement method for the multimodal generative sentiment analysis model according to claim 1, characterized in that: The target transformation relationship includes: The consistent target transformation relationship under sentence order changes, the consistent target transformation relationship under key sentence relocation, the consistent target transformation relationship under entity replacement, and the inconsistent target transformation relationship under modifier reversal.

3. The robust enhancement method for the multimodal generative sentiment analysis model according to claim 1, characterized in that: The initial multimodal generative sentiment analysis model is a multi-layer attention network; The multimodal generative sentiment analysis model is trained based on the training set and the sentiment labels corresponding to the training set to obtain a new multimodal generative sentiment analysis model, including: Inputting the text in the training set into the task paraphrase multilayer perceptron to obtain a new text set, and performing feature extraction on the new text set to obtain text features; Adding continuous prompts to the images in the training set to obtain a new image set, and performing feature extraction on the new image set to obtain image features; A multi-layer attention network is trained based on the text features, the image features and the corresponding sentiment labels to obtain a new multimodal generative sentiment analysis model.

4. The robust enhancement method for the multimodal generative sentiment analysis model according to claim 3, characterized in that: A multi-layer attention network is trained based on the text features, the image features and the corresponding sentiment labels to obtain a new multimodal generative sentiment analysis model, including: Inputting the text features and the image features into a multi-layer attention network to train the multi-layer attention network, and during the training process, cross-attentionally fusing the text modality features and the image modality features output by the multi-layer attention network to obtain multi-modal fusion features; Based on the multimodal fusion feature, determining the cross entropy loss between the predicted emotion label and the real emotion label corresponding to the text feature and the image feature; Based on the cross entropy loss, the trained multi-layer attention network is optimized to obtain a new multimodal generative sentiment analysis model.

5. The robust enhancement method for the multimodal generative sentiment analysis model according to claim 4, characterized in that: Determining the cross entropy loss between the predicted emotion label and the real emotion label corresponding to the text feature and the image feature based on the multimodal fusion feature includes: Inputting the hidden state of the multimodal fusion feature into a multi-layer perceptron to obtain a predicted score probability; Based on the predicted score probability, a cross entropy loss between the predicted emotion label and the true emotion label corresponding to the text feature and the image feature is calculated.

6. The robust enhancement method for a multimodal generative sentiment analysis model according to claim 1, characterized in that: Based on the original input set and the target transformation relationship, a new input set is constructed, including: In the case where the original input set is text, generating seed prompts and keywords of the original input set based on the task description; deriving the most task-relevant paraphrase cue from the seed cue; Based on the paraphrase prompt, the keyword and the original input set, and in combination with the target metamorphosis relationship, a new input set is constructed.

7. The robust enhancement method for a multimodal generative sentiment analysis model according to any one of claims 1 to 6, characterized in that: Based on the original input set and the target transformation relationship, a new input set is constructed, including: In the case where the original input set is an image, the original input set is divided into a plurality of regions, and a vector is initialized for each region to obtain a series of continuous vectors; Based on the continuous vector and the target transformation relationship, a new input set is constructed.

8. The robust enhancement method for a multimodal generative sentiment analysis model according to claim 7, characterized in that: Based on the continuous vector and the target transformation relationship, a new input set is constructed, including: An initial vector based on the image embedding in the original input set is used as a prefix of the continuous vector; A new input set is constructed based on the continuous vector and the prefix of the continuous vector and in combination with a target metamorphosis relationship.

9. A multimodal sentiment classification method, characterized in that: include: Obtain the data set to be identified; The data set to be identified is input into the multimodal generative sentiment analysis model obtained by the method according to any one of claims 1 to 8 to obtain the sentiment type.

10. A robust enhancement device for a multimodal generative sentiment analysis model, characterized in that: include: A data acquisition module, used to acquire an original input set, and construct a new input set based on the original input set and a target metamorphosis relationship; The original input set includes text and images; A label extraction module, used for inputting the original input set and the new input set into the initial multimodal generative sentiment analysis model respectively to obtain sentiment labels; A training set construction module, used for selecting expected data from the original input set and the new input set based on the emotion label to construct a training set; The model training module is used to train the multimodal generative sentiment analysis model based on the training set and the sentiment tags corresponding to the training set to obtain a new multimodal generative sentiment analysis model.

Images