Multi-modal natural dialogue context sentiment analysis method based on personalized sentiment feature generation

Abstract

The invention particularly relates to a multi-modal natural dialogue context sentiment analysis method based on personalized sentiment feature generation. The method comprises the steps of obtaining a training set; constructing a multi-modal sentiment analysis model for performing feature fusion through a cross-modal attention mechanism based on input visual, sound and text three-modal features, outputting a sentiment classification result, and calculating clustering center features of each sentiment type; generating personalized multi-modal features of the specific speaker by using the initialization information; training the multi-modal emotion analysis model by using the training set, performing parameter fine tuning on the model in the training stage by using the personalized multi-modal features of the specific speaker, and optimizing the emotion recognition ability for the specific speaker; and applying the fine-tuned multi-modal sentiment analysis model to the new dialogue data, and outputting a final sentiment classification result. According to the method, the personalized adaptive process related to the speaker is combined with the universal multi-modal sentiment analysis model, so that the overall recognition efficiency and precision are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of user personalized emotion recognition, and specifically relates to a multi-modal natural dialogue context emotion analysis method based on personalized emotion feature generation. Background Art

[0002] The multi-modal natural dialogue context emotion analysis technology refers to a method for realizing multi-modal emotion type recognition by analyzing, fusing and calculating the multi-modal features of the speaker in the natural interaction scenario between humans and machines. With the development of deep learning technology, the application of artificial intelligence-related technologies has been continuously popularized and implemented, and the human-computer interaction ability in the actual use process has gradually become a new research hotspot. Natural dialogue not only conforms more to the daily communication habits of humans, but also can break through the barriers of limited instruction set interaction and expand the new boundaries of human-computer interaction. As one of the key basic research contents in human-computer interaction, emotion computing has broad development potential in multiple directions such as commerce, people's livelihood, and military industry. For example: the current digital employee plan, mechanical equipment control, security monitoring in important places, medical all-weather escort, smart home assistant and mental health analysis, etc.

[0003] Under natural context conditions, existing multi-modal emotion analysis technologies mostly use three signals, namely vision, sound and text, as input modalities for calculation, and process and fuse multi-modal features through a deep neural network. The current technologies related to the multi-modal emotion analysis of the speaker can be mainly divided into three research directions according to the different emphases of speaker feature recognition: speaker recognition, generalization ability enhancement, and personalized feature recognition.

[0004] Among them, speaker recognition mainly enhances the adaptability of the multi-modal emotion analysis network to different speakers by adding a speaker recognition module in parallel with the original multi-modal emotion recognition network. However, this method focuses on speaker recognition and cannot directly enhance the extraction and fusion capabilities of multi-modal emotion features. The enhancement of generalization ability mainly optimizes the structure or feature processing algorithm of the multi-modal emotion analysis model to achieve the de-differentiation of emotion features of different speakers and extract common features in the same emotion type. However, this method is highly dependent on algorithm and network structure design, and the distribution and comprehensiveness of data sources will also have a great impact on the model. Finally, personalized feature recognition is a new requirement and direction that emerged with the implementation of artificial intelligence productization in recent years. When artificial intelligence interacts with users in practice, the ability to understand and learn the personalized features of specific users has become one of the important factors in improving product service capabilities and user experience. However, most current multi-modal emotion analysis algorithms mainly focus on adding a speaker recognition module and enhancing generalization recognition ability. For the learning of speakers' personalized features, they mainly add parallel emotion type recognition channels, optimize the loss function, or fine-tune specific difficult emotion recognition types, and cannot truly achieve dynamic learning and fitting of the emotion features of specific speakers, lacking the ability to improve the emotion recognition accuracy for actual speakers or fixed users, thus hindering the targeted real-time analysis and improvement of human-computer interaction according to user characteristics in actual natural dialogue scenarios.

[0005] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present invention, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0006] The present invention provides a multi-modal natural dialogue context emotion analysis method based on the generation of personalized emotion features, which can achieve better multi-modal emotion analysis capabilities when facing a known specific speaker, and thus can overcome the defects existing in the prior art to a certain extent.

[0007] Other features and advantages of the present invention will become apparent through the following detailed description, or will be partially learned through the practice of the present invention.

[0008] According to a first aspect of the present invention, there is provided a multi-modal natural dialogue context emotion analysis method based on the generation of personalized emotion features, the method comprising:

[0009] Separate the multi-modal data in the natural dialogue by speaker to generate a multi-modal dialogue feature set for each speaker, use the first two sentences as initialization information, and the subsequent dialogue features as generation targets to construct a training set including initialization information, target features, and emotion labels;

[0010] Construct a multi-modal sentiment analysis model based on the Transformer neural network. The multi-modal sentiment analysis model is used to perform feature fusion through a cross-modal attention mechanism based on the input visual, audio, and text multi-modal features, output a sentiment classification result, and calculate the clustering center features of each sentiment type.

[0011] Based on a personalized multi-modal feature data generation model, use the initialization information to generate personalized multi-modal features of a specific speaker.

[0012] Use the training set to train the multi-modal sentiment analysis model, and use the personalized multi-modal features of a specific speaker to fine-tune the parameters of the model during the training phase to optimize the sentiment recognition ability for the specific speaker.

[0013] Apply the fine-tuned multi-modal sentiment analysis model to new conversation data and output the final sentiment classification result.

[0014] In some exemplary embodiments, the multi-modal sentiment analysis model includes an input layer, a normalization layer, a Transformer layer, a first fully connected layer, an activation layer, a second fully connected layer, and an output layer; input the visual, audio, and text multi-modal features of the input conversation statement, and the input layer performs concatenation processing on the multi-modal features; then extract and process the features through the normalization layer and the Transformer layer; finally, output the recognition result through the first fully connected layer, the activation layer, the second fully connected layer, and the output layer, and calculate the clustering center features of each sentiment type through the first fully connected layer.

[0015] In some exemplary embodiments, the specific implementation of the cross-modal attention mechanism is as follows:

[0016] Use the text modal features as the query vector to assign attention weights to the image and audio modal features. The formula is:

[0017] La A = Trans(A|T)

[0018] La V = Trans(V|T)

[0019] La T = Trans(T)

[0020] After concatenating the three-modal hidden vectors, output the sentiment classification result through the fully connected layer.

[0021] In some exemplary embodiments, the personalized multi-modal feature data generation model includes an input layer, a generative diffusion model, and an output layer; the input layer cascades the image modality features in the initialization statement features and inputs them into the generative diffusion model; the text and voice modality dialogues are cascaded within the dialogue and then cascaded between dialogues and input into the generative diffusion model, and the generative diffusion model generates a three-modal feature result and outputs it through the output layer.

[0022] In some exemplary embodiments, the generative diffusion model is implemented based on a U-net structure, and the emotional clustering center features are used as auxiliary conditions during the generation process.

[0023] In some exemplary embodiments, the generative diffusion model includes a continuous generative diffusion model and a discrete generative diffusion model. Among them, the continuous generative diffusion model is used for the image modality, and the discrete generative diffusion model is used for the text and voice modalities.

[0024] In some exemplary embodiments, the loss functions involved in the training process include a first loss function, a second loss function, and a third loss function:

[0025] The first loss function is the cross-entropy between the recognition result output during the training stage and the emotion label;

[0026] The second loss function is the mean absolute error between the multi-modal features generated during the data augmentation stage and the true generation features fitting the multi-modal features; the similarity loss between the generated features and the true features.

[0027] The third loss function is the cross-entropy between the recognition result output during the fine-tuning stage and the emotion label.

[0028] According to the second aspect of the present invention, there is provided a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the multi-modal natural dialogue context emotion analysis method based on personalized emotion features described in the first aspect above.

[0029] According to the third aspect of the present invention, there is provided a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, it implements the multi-modal natural dialogue context emotion analysis method based on personalized emotion features described in the first aspect above.

[0030] According to the fourth aspect of the present invention, there is provided an electronic device, including:

[0031] A processor; and

[0032] A memory for storing executable instructions of the processor;

[0033] Among them, the processor is configured to implement the multi-modal natural dialogue context sentiment analysis method based on personalized sentiment features described in the first aspect above when executing the executable instructions.

[0034] The multi-modal natural dialogue context sentiment analysis method based on personalized sentiment features provided by the embodiments of the present invention uses a generative diffusion model to generate data for personalized multi-modal sentiment features related to the speaker, and realizes the adaptive adjustment of the sentiment recognition model for the speaker in the case of insufficient speaker personalization features. Compared with other methods that completely rely on large-scale training sets to optimize model parameters and construct a general multi-modal sentiment analysis model, the combination of the personalized adaptive process related to the speaker and the general multi-modal sentiment analysis model improves the overall recognition efficiency and accuracy.

[0035] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. Brief Description of the Drawings

[0036] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0037] Figure 1 is the structural diagram of the multi-modal sentiment analysis network;

[0038] Figure 2 is the structural diagram of the personalized multi-modal feature data generation network;

[0039] Figure 3 is the overall network structural diagram of the multi-modal sentiment analysis method based on personalized sentiment features;

[0040] Figure 4 is the flowchart of the method of the embodiment of the present invention;

[0041] Figure 5 is the test result diagram: (a) is the recognition result of the multi-modal sentiment analysis model before the model parameters are further fine-tuned based on the augmented data; (b) is the sentiment classification result of the multi-modal sentiment analysis model after the model parameters are further fine-tuned based on the augmented data. Detailed Embodiments

[0042] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments.

[0043] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in the form of software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0044] In the related art, in order to extract the emotional voice timbre of a speaker, based on speaker recognition technology, transfer learning is used to fine-tune a network according to the collected emotional speech features of the speaker, so as to extract speaker features with emotional information. This method has achieved good recognition results on related data sets and can better compensate for and synthesize the emotional voice timbre of the speaker. However, this method mainly uses existing data in speaker emotion recognition, performs functional fine-tuning based on the speaker recognition model, and mainly extracts low-level handcrafted features such as Mel cepstral coefficients, etc., without involving new speaker feature learning, nor is it verified on an unknown speaker data set, ignoring the influence of individual feature differences between speakers on emotion recognition.

[0045] Most existing multi-modal sentiment analysis models rely on neural networks to automatically extract, select, and fuse high-dimensional heterogeneous features in the latent vector space, without paying attention to personalized features closely related to the speaker himself. On the contrary, most existing methods are committed to extracting common sentiment features that are not affected by factors such as the speaker's own expression habits and cultural customs, and try to achieve the purpose of removing the characteristics of the speaker himself, so as to establish a more widely applicable general sentiment analysis model. For this research shortcoming, the present invention proposes for the first time the concept of human-computer interaction and the vision of sentiment analysis of "the more interactive, the more understanding" for the multi-modal sentiment analysis method in the natural dialogue scenario with a low speaker replacement frequency. By introducing a generative diffusion model, data fitting, generation, and multi-modal feature augmentation of the emotional expression characteristics of a specific speaker are carried out to solve the cold start problem of emotion recognition in the actual human-computer interaction process, with a view to quickly learning the personalized emotional characteristics of the speaker, and thus completing the internal targeted fine-tuning of the multi-modal sentiment analysis model based on transformer, and improving the emotion analysis accuracy for specific users and the adaptive learning ability of the model. The specific process is as follows: First, build a multi-modal sentiment analysis model based on the transformer neural network and train a general sentiment analysis model that does not distinguish speakers; then, use the generative diffusion model to fit the data of the limited and possibly repeated multi-modal sentiment features of the speaker and generate personalized features under each emotion type, providing a data basis for the subsequent fine-tuning of the sentiment analysis model; finally, based on the augmented multi-modal data related to the speaker, fine-tune the parameters of the model based on transformer to obtain the final recognition result. The proposed method combines the deep neural network based on transformer with the generative diffusion model, and redesigns the algorithm structure, steps, and loss function composition, thus effectively improving the recognition accuracy and efficiency.

[0046] In order to have better multi-modal sentiment analysis capabilities when facing a known specific speaker, the present invention proposes a multi-modal natural dialogue context sentiment analysis method based on the generation of personalized sentiment features. Using three modalities, namely vision, sound, and text, as input and analysis objects, it is divided into five stages: a preprocessing stage, a training stage, a data augmentation stage, a fine-tuning stage, and a testing stage. The specific implementation process is as follows: First, design a multi-modal sentiment analysis model based on the transformer neural network to train a general sentiment analysis model without distinguishing speakers; then, based on the generative diffusion model, perform data fitting on the personalized multi-modal sentiment features of the speaker and generate multi-modal features corresponding to each sentiment type, thereby completing the augmentation of multi-modal feature data corresponding to the speaker under each sentiment type and providing a data basis for the subsequent fine-tuning of the sentiment analysis model; finally, based on the augmented multi-modal data related to the speaker, perform parameter fine-tuning on the transformer-based model, and use the fine-tuned model to identify the new dialogue information of the speaker to obtain the final identification result.

[0047] In the multi-modal dialogue sentiment recognition technology method of the present invention, in order to use the generative diffusion model to perform data fitting and augmentation on the personalized multi-modal sentiment features of the speaker, it is necessary to preprocess the original dataset without changing the corresponding features of the original sentences. The specific method is as follows: First, in the same natural language dialogue, separate the dialogue features according to different speakers and still arrange them in the order of dialogue occurrence to generate a multi-modal dialogue feature set corresponding to each speaker; then, use the first two sentences in the multi-modal dialogue feature set of each speaker as initialization information, and the subsequent dialogue features as the generation and fitting objects to generate a new training set and test set for the second stage to perform data fitting on the personalized multi-modal sentiment features of the speaker and generate multi-modal features corresponding to each sentiment type. In the new training set and test set, each piece of data contains two sentences of initialization information and one sentence to be fitted, as well as the sentiment type labels corresponding to the three sentences.

[0048] In the training stage, design a sentiment analysis model for the three modalities of image, sound, and text based on the transformer, mainly to complete the extraction of three-modal features and the fusion of modal features based on the transformer, and output the result of sentiment classification. Among them, when performing single-modal feature encoding on the image and speech modalities, introduce the text vector as the query value, and use the text modality as the central modality to provide guidance for the single-modal feature encoding process of the image and speech modalities. The training stage uses the original dataset without preprocessing for training, and after training, use K-means to obtain the distribution of feature clustering centers of each modality under each sentiment type. The trained model will be further fine-tuned after completing data augmentation in the subsequent stage.

[0049] In the data augmentation stage, based on the new dataset obtained in the preprocessing stage, by generating a diffusion model and the speaker initialization information in each piece of data, personalized generation of the multimodal features of the speaker is performed. The specific method is as follows: Use a continuous generation diffusion model to fit and generate the image modality features of the speaker, and based on a discrete generation diffusion model, perform feature fitting and generation on the text modality and voice modality of the speaker. The generated features are compared with the true emotional multimodal features of the speaker in this piece of data, so as to generate negative feedback constraints until the training of the generation model is completed, and then the multimodal features generated under each emotional type of the corresponding speaker are obtained, which are used as augmented data for fine-tuning the emotion analysis model established in the training stage.

[0050] In the fine-tuning stage, use the newly generated multimodal emotional features related to the speaker under each emotional type obtained in the data augmentation stage to further fine-tune the model parameters of the well-trained multimodal emotion analysis model in the training stage.

[0051] In the testing stage, use the new test set data, that is, the new dialogue data of the speaker, to test the fine-tuned multimodal emotion analysis model, and complete the output of the final emotion type recognition result.

[0052] The loss function part is mainly composed of two parts: the classification error in the training stage and the fine-tuning stage, and the similarity error in the data augmentation stage.

[0053] In the present invention, the network structure of the multimodal emotion analysis method based on personalized emotion feature generation is divided into two parts: a multimodal emotion analysis network and a personalized multimodal feature data generation network. Among them, for the structure diagram of the multimodal emotion analysis network, please refer to Figure 1 For the structure diagram of the personalized multimodal feature data generation network, please refer to Figure 2 For the overall model network structure diagram, please refer to Figure 3 .

[0054] In the present invention, for the network structure of the multimodal emotion analysis network, please refer to Figure 1 It has a total of 6 layers. Among them, there is 1 output layer, 1 input layer, and 5 hidden layers, including 1 normalization layer, 1 transformer layer, 2 fully connected layers, and 1 activation layer. First, input the image, voice, and text three-modal features of the dialogue statement. The input layer performs concatenation processing on the three-modal features, and then extracts and processes the features through normalization and the transformer. Finally, the recognition result is output through the fully connected layer and the activation layer, and the multimodal emotion features after clustering under each emotion type are counted. Next, for the structure of the personalized multimodal feature data generation network, please refer to Figure 2, which is divided into three parts, namely the input layer part, the generative diffusion model part, and the output part. The input layer cascades the image modality features in the initialization statement features, and then cascades the text and voice modality dialogues within and between dialogues. After that, the image modality passes through the continuous generative diffusion model, and the text and voice modalities pass through the discrete generative diffusion model for generation and processing respectively. Both generative diffusion models are built based on the U-net model structure. Finally, the generated three-modal feature results are output. Finally, for the overall model network structure, please refer to Figure 3 , use the generated personalized multi-modal data to fine-tune the trained multi-modal sentiment analysis network, and use the fine-tuned model to output the multi-modal sentiment recognition results on the final test set.

[0055] Compared with the existing multi-modal sentiment recognition models, in the model of the present invention, by introducing the generative diffusion model, the multi-modal sentiment data of the speaker is augmented based on the speaker's initialization information, changing the previous mode that focused on developing general sentiment recognition models to the continuous optimization and improvement of recognition accuracy for the speaker, so as to achieve a fast adjustment and adaptive process for a specific speaker in an actual natural dialogue scenario.

[0056] Refer to Figure 4 As shown, the embodiment of the present invention provides a multi-modal sentiment analysis method based on personalized sentiment feature generation. With the real classification labels being six basic sentiment types (happy, excited, frustrated, sad, angry, and neutral), a method combining the generative diffusion model and the transformer neural network is adopted to realize the further optimization process of fine-tuning the model parameters according to the speaker's personalized features, and complete the multi-modal sentiment recognition and optimization of the target speaker. Specifically, it may include the following steps:

[0057] Step 1: Build a multi-modal natural dialogue context sentiment analysis algorithm model based on personalized sentiment feature generation, and train the model using the gradient descent and backpropagation algorithms. The specific process is as follows:

[0058] Step 1-1: In the training stage, according to Figure 1 Build a multi-modal sentiment analysis network algorithm model based on the transformer neural network, and initialize all parameters and weights with random numbers. The three-modal feature vectors of the used images, voices, and texts are respectively represented as: A, V, T. The three groups of hidden vector expressions obtained after the multi-modal features pass through the transformer are respectively denoted as La A , La V and La T , and the formula is:

[0059] La A = Trans(V|T)

[0060] La V = Trans(V|T)

[0061] La T = Trans(T)

[0062] Among them, La A and La V are both performed under the condition that the text modality T is the query vector. Concatenate the three groups of hidden vectors, and obtain the preliminary sentiment recognition result C through two fully connected layers Linear1, Linear2 and an activation layer tanh init .

[0063] C init = Linear2(tanh(Linear1(La A , La V , La T )))

[0064] Next, obtain the clustering center eigenvalues La of the sentiment features after multimodal fusion before the output layer through Kmeans center .

[0065] L acenter = Kmeans(Linear1(La A , La V , La T ))

[0066] Step 1-2, data augmentation stage. In the same natural language conversation, separate the conversation features according to different speakers and still arrange them in the order of conversation occurrence to generate the multimodal conversation feature set corresponding to each speaker; then, use the first two sentences in the multimodal conversation feature set of each speaker as the initialization information, and the subsequent conversation features as the generation and fitting objects to generate new training and test sets. The information contained in each piece of data is recorded as Item, where:

[0067] Item = [Initial1(A1, V1, T1), Initial2(A2, V2, T2), Target(A t , V t , T t ), C T

[0068] Among them, Initial1(A1, V1, T1) and Initial2(A2, V2, T2) respectively represent the initialization information for new data generation. Both of them come from the same speaker and both contain the three-modal feature information of sound, text and image of the corresponding sentences. Target(A t , V​t ,T t ), where C is the corresponding multi-modal information to be fitted T is the emotion type corresponding to the multi-modal information to be fitted

[0069] Next, according to Figure 2 , the image modality features of the initial information are cascaded, denoted as V concat , and the feature generation is performed through the continuous generation diffusion model D based on U-net cont . After the sound and text modalities of the initial information are intra-sententially cascaded respectively and then inter-sententially cascaded, it is denoted as AT concat , and the feature generation is performed through the discrete generation diffusion model D disc . Among them, La corresponding to C T is used as the auxiliary conditional information. The generated image modality V center and the sound modality A gen , and the text modality T gen are respectively: gen are respectively:

[0070] V gen = D cont (V concat |La center )

[0071] A gen ,T gen = D disc (AT concat |La center )

[0072] Target’(A t ,V t ,T t ) = [A gen ,V gen ,T gen

[0073] Step 1-3, Fine-tuning and Testing Phase. According to Figure 3 , use the initial information in the test dataset to generate the new image modality V gen , sound modality A gen and text modality T gen of the speaker corresponding to the initial information. Use these generated pseudo-speaker features to further train and fine-tune the parameters of the multi-modal emotion analysis network algorithm model based on the transformer neural network built in the training phase, and complete the parameter fine-tuning of the personalized emotion features of the speakers corresponding to Initial1(A1, V1, T1) and Initial2(A2, V2, T2).

[0074] ​After the fine-tuning is completed, perform emotion recognition on the real speaker test data to complete the final actual emotion recognition process. The finally recognized emotion type is denoted as C gen 。

[0075] Next, calculate the loss function. The loss function consists of three parts. One part is the recognition result C output during the training phase init and the cross-entropy Loss between the emotion label Label CE :

[0076] Loss CE =CrossEntrophy(C init ,Label)

[0077] The second part of the loss function is Loss rec , which is the mean absolute error between the generated multimodal feature Target’(A t ,V t ,T t ) and the real proxy multimodal feature Target(A t ,V t ,T t ). MAE is the formula for calculating the mean absolute error:

[0078] Loss rec =MAE(Target’(A t ,V t ,T t ),Target(A t ,V t ,T t ))

[0079] The third part is the cross-entropy Loss between the recognition result C output during the model parameter fine-tuning period and the emotion label Label tune : CE-finetune :

[0080] Loss CE =CrossEntropyh y(C tune ,Label)

[0081] The three parts of the loss function are calculated and backpropagated respectively during their respective training phases, and finally the training of the model is completed.

[0082] Step 2: Use the data that has not been trained in the dataset as the test set, and use the multimodal natural dialogue context emotion analysis algorithm model generated based on personalized emotion features for calculation to obtain the final classification result.

[0083] Among them, Step 1 is the construction of the main model, and Step 1 is the inventive point of the present invention.

[0084] The technical effects of the present invention will be described in detail below in combination with simulation experiments.

[0085] The present invention has been verified for effectiveness on the multi-modal sentiment analysis public dataset IEMOCAP. The training set of the IEMOCAP dataset contains 5,450 samples, and the test set contains 1,530 samples, meeting the training-test ratio of 3:1. Each sample contains text, image, and speech information, and the labels are divided into six categories, namely happy, excited, frustrated, sad, angry, and neutral. The evaluation metric is F1-score, and a significance test is conducted through a T-test with a significance level of 0.05 and a learning rate of 0.0015. The test results, the comparison chart of the model parameters before and after fine-tuning, are as Figure 5 shown in the confusion matrix in which the F1-score is 0.80, proving the effectiveness of the method described in the present invention.

[0086] It should be noted that, on the other hand, the present application also provides a storage medium, which may be included in an electronic device; or it may exist alone without being assembled into the electronic device. The above storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device is caused to implement the method described in the following embodiments.

[0087] In one embodiment, the present application provides a computer program product, including a computer program, which when executed by a processor implements the steps in the above method embodiments.

[0088] In addition, the above drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0089] Those skilled in the art will readily think of other embodiments of the present invention after considering the specification and practicing the invention herein. The present application aims to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include the common general knowledge or conventional technical means in the technical field not disclosed in the present invention. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present invention are pointed out by the claims.

[0090] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only defined by the appended claims.

Claims

1. A multi-modal natural dialogue context sentiment analysis method based on personalized emotion features, characterized in that, The method includes: Separating the multimodal data in the natural conversation by speaker, generating a multimodal dialogue feature set for each speaker, using the first two sentences as initialization information, and the subsequent dialogue features as generation targets, and constructing a training set including initialization information, target features, and sentiment labels; Constructing a multimodal sentiment analysis model based on the Transformer neural network, which is used to perform feature fusion through a cross-modal attention mechanism based on the input visual, audio, and text triple-modal features, output a sentiment classification result, and calculate the clustering center features of each sentiment type; Based on the personalized multimodal feature data generation model, using the initialization information to generate personalized multimodal features of a specific speaker; Using the training set to train the multimodal sentiment analysis model, and using the personalized multimodal features of a specific speaker to fine-tune the parameters of the model during the training stage to optimize the sentiment recognition ability for a specific speaker; Applying the fine-tuned multimodal sentiment analysis model to new dialogue data and outputting the final sentiment classification result.

2. The method according to claim 1, wherein The multimodal sentiment analysis model includes an input layer, a normalization layer, a transformer layer, a first fully connected layer, an activation layer, a second fully connected layer, and an output layer; the image, audio, and text triple-modal features of the input dialogue statement are cascaded by the input layer; then the features are extracted and processed by the normalization layer and the transformer layer; finally, the recognition result is output through the first fully connected layer, the activation layer, the second fully connected layer, and the output layer, and the clustering center features of each sentiment type are calculated through the first fully connected layer.

3. The method according to claim 1, wherein The specific implementation of the cross-modal attention mechanism is: Using the text modal features as query vectors to allocate attention weights to the image and audio modal features, and the formula is: La A = Trans(A|T) La V = Trans(V|T) La T = Trans(T) The triple-modal hidden vectors are cascaded and then output the sentiment classification result through the fully connected layer.

4. The method according to claim 1, wherein The personalized multimodal feature data generation model includes an input layer, a generative diffusion model, and an output layer; the input layer cascades the image modal features in the initialization statement features and inputs them into the generative diffusion model; the text and audio modal dialogues are cascaded within the dialogue and then cascaded between dialogues and input into the generative diffusion model, and the generative diffusion model generates triple-modal feature results and outputs them through the output layer.

5. The method according to claim 4, characterized in that, The generative diffusion model is implemented based on the U-net structure, and the sentiment clustering center features are used as auxiliary conditions during the generation process.

6. The method according to claim 5, wherein The generative diffusion model includes a continuous generative diffusion model and a discrete generative diffusion model. Among them, the continuous generative diffusion model is used for the image modality, and the discrete generative diffusion model is used for the text and audio modalities.

7. The method according to claim 1, characterized in that, The loss functions involved in the training process include a first loss function, a second loss function, and a third loss function: The first loss function is the cross-entropy between the recognition result output during the training stage and the sentiment label; The second loss function is the mean absolute error between the multimodal features generated during the data augmentation stage and the real proxy fitting multimodal features, the similarity loss between the generated features and the real features; The third loss function is the cross-entropy between the recognition result output during the fine-tuning stage and the sentiment label.

8. A storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by a processor, it implements the multi-modal natural dialogue context sentiment analysis method based on personalized sentiment features described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the multi-modal natural dialogue context sentiment analysis method based on personalized sentiment features described in any one of claims 1 to 7.

10. An electronic device, characterized in that, Comprising: A processor; And A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the multi-modal natural dialogue context sentiment analysis method based on personalized sentiment features described in any one of claims 1 to 7 by executing the executable instructions.

Images