Knowledge-enhanced product question-answering community user conversation emotion recognition method and system

Abstract

The present invention provides a method, system, storage medium, and electronic device for recognizing emotions in user conversations in a knowledge-enhanced product question-and-answer community, relating to the field of conversation emotion recognition. The present invention constructs a user-personalized emotion dictionary through user keyword extraction and dictionary expansion algorithms to obtain a preference knowledge embedding vector. Through a cross-modal Transformer, the emotion-enhanced conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector are integrated to obtain a knowledge-enhanced conversation feature vector. Based on a conversation context GCN network, the global context vector is obtained by considering the context in multiple rounds of question-and-answer information and the contextual relationships between conversation users. Personalized emotion recognition is performed by considering the interaction between the global context vector and the conversation feature vector. Furthermore, a two-layer supervised comparative learning strategy based on the emotion layer and the multimodal layer is proposed to enable the model to learn more discriminative multimodal feature representations, further improving the accuracy and granularity of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of conversation emotion recognition, and in particular to a method, system, storage medium and electronic device for recognizing conversation emotions among users of a knowledge-enhanced product question-and-answer community. Background Art

[0002] In recent years, with the rapid development of internet technology, product Q&A communities have become a vital platform for knowledge sharing and user interaction. The conversations and emotional expressions between questioners and answerers in product Q&A communities reflect user satisfaction with the quality of answers. Providing users with satisfactory answers promotes knowledge sharing, improves community information quality, and maintains the healthy development of the community.

[0003] In related technologies, Q&A community platforms monitor and analyze user emotions in real time. If users express dissatisfaction, the platform promptly invites technical experts in the field to provide professional answers and support. Furthermore, the platform dynamically adjusts its response strategy based on the results of sentiment analysis, aiming to proactively respond to and effectively address user needs. This strategy can improve user satisfaction, help the platform build a positive image, and thus attract more potential users to participate in community interactions and discussions, further expanding the community's scale and influence.

[0004] However, existing solutions are usually generalized models. Considering that different users have different perceptions and expressions of emotional information, it is necessary to provide a new emotion recognition technology solution to achieve personalized emotion recognition. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the deficiencies in the existing technology, the present invention provides a knowledge-enhanced method, system, storage medium, and electronic device for identifying emotions in user conversations in a product Q&A community, which solves the technical problem of ignoring that different users have different perceptions and expressions of emotional information.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A knowledge-enhanced method for identifying emotions in user conversations in product question-and-answer communities, based on a pre-set model, includes:

[0010] Obtaining historical multi-round question-and-answer information from multiple product question-and-answer communities; each of the historical multi-round question-and-answer information includes a user's multimodal conversation data and its emotion label, and each piece of the multimodal conversation data includes text, visual, and audio modal information;

[0011] For the current multi-round question-and-answer information, obtain the conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector corresponding to each piece of the multimodal conversation data;

[0012] For any piece of multimodal conversation data, the similarity between each original word in its text modal information and a keyword in a pre-built user-personalized sentiment dictionary is calculated, the embedding representation of each original word is concatenated with the preference knowledge embedding vector corresponding to the keyword with the highest similarity, and the corresponding highest similarity is used as a weight to multiply the concatenated result to obtain a knowledge-enhanced conversation text embedding vector;

[0013] Taking the knowledge-enhanced conversation text embedding vector as the main modality, and fusing it with the conversation visual embedding vector and the conversation audio embedding vector using a cross-modal Transformer to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data;

[0014] Based on the conversation feature vector, an LSTM network is used to model the contextual emotional dependency of users in each piece of the multimodal conversation data to obtain a contextual situation vector for personalized preference knowledge perception; and based on the contextual situation vector, a global conversation graph is constructed, and a GCN network is used to model the emotional dependency between users in each piece of the multimodal conversation data to obtain a global situation vector that stores a global context state.

[0015] Obtaining an emotion recognition prediction result for each piece of the multimodal conversation data through a collaborative attention mechanism based on the interaction between the conversation feature vector and the global context vector;

[0016] Constructing a contrastive learning loss based on a dual-layer supervised contrastive learning strategy of the sentiment layer and the multimodal layer according to the conversation text embedding vector, the conversation visual embedding vector, and the conversation audio embedding vector corresponding to each piece of the multimodal conversation data; constructing a cross-entropy loss based on the emotion label and emotion recognition prediction result of each piece of the multimodal conversation data; minimizing a joint loss function of the contrastive learning loss and the cross-entropy loss, and training the model until convergence;

[0017] The multi-round question-and-answer information to be recognized is used as the input of the converged model to obtain the emotion recognition results of each multimodal dialogue data in the multi-round question-and-answer information.

[0018] Preferably, the process of obtaining contrastive learning loss based on sentiment layer includes:

[0019] Map the conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector into the three-dimensional VAD emotion space:

[0020]

[0021] Among them, the modality m∈{T,V,A}, T,V,A are the text, visual and audio modal information in the multimodal conversation data respectively; is the corresponding multimodal embedding vector, is the feature representation mapped to the VAD emotion space, i is the index of the multimodal conversation data; R is the set; W is the weight parameter; b is the bias term;

[0022] Introducing manually labeled e for each emotion in the VAD emotion space c ∈R 3 ; where c∈E represents the emotion category, E represents the set of emotion label categories, e c Represents the VAD vector corresponding to the true emotion label;

[0023] Each piece of multimodal conversation data is taken as a sample. The feature representation of the three modal information in each sample is mapped to the VAD emotion space. The VAD vector corresponding to its true emotion label is regarded as a positive sample pair, and the VAD vector of other types of emotions is regarded as a negative sample pair. The first contrastive learning loss is constructed:

[0024]

[0025] Where |E| is the number of emotion categories; the hyperparameter λ m Used to adjust the weight of the loss function for the consistency comparison between each modality information and the label; log is the logarithmic function; Represents the true emotion label y of the i-th multimodal dialogue data i The corresponding VAD vector; represents the dot product operation, which is used to calculate the similarity between vectors; τ is a learnable hyperparameter.

[0026] Preferably, the process of obtaining a contrastive learning loss based on a multimodal layer comprises:

[0027] The multimodal embedding vectors of samples with the same emotion are regarded as positive sample pairs, and the multimodal embedding vectors of samples with different emotions are regarded as negative sample pairs, and the second contrastive learning loss is constructed:

[0028]

[0029] Among them, m1 and m2 are two arbitrary modes; λ m1,m2 N is the weight hyperparameter for contrast learning between different modalities; d is the number of samples of previous multi-round question-answering information; The label is y in the current historical multi-round question and answer information i The number of samples; j, k are the indexes of the multimodal conversation data used as samples.

[0030] Preferably, the process of constructing the user personalized sentiment dictionary includes:

[0031] The manually annotated sentiment dictionary NRC-VAD Chinese version is selected as the basic sentiment dictionary, which is expressed as Where (V i , A i , D i ) is the word w i The features of represent the corresponding VAD emotional state vector, which respectively represent the scores of activation, pleasure and dominance in three dimensions as V i , A i , D i , and the value range of the three is between 0 and 1;

[0032] Crawl the historical Q&A information of all users in the current product Q&A community, build a Q&A corpus for the target domain, remove stop words to obtain a domain dictionary W, and use this dictionary to train a Word2Vec model to map each word in W to a high-dimensional vector containing the word's semantic information. Use cosine similarity to measure the semantic similarity between any two words.

[0033] For words in W, either belonging to the dictionary L0 or not belonging to the basic sentiment dictionary L0, through the semantic similarity between any two words and several similar words, if there is a similar word belonging to the dictionary L0, then the word is regarded as a potential sentiment word and added to the potential sentiment word set P;

[0034] For each emotion word in the set P, define the algorithm f to map its feature representation into a VAD emotion state vector; if the three dimensions of the VAD emotion state vector are all greater than 0, then add the emotion word and the corresponding VAD emotion state vector to the dictionary L0, and record the expanded domain emotion dictionary as L new ;

[0035] Collect the current user's historical question and answer information and treat it as a document. Remove stop words from the document and then perform word segmentation.

[0036] Use TF-IDF method to extract all keywords from preprocessed documents;

[0037] If the keyword is in the expanded domain sentiment dictionary L new Then find the VAD emotional state vector corresponding to the keyword; if the keyword is not in the expanded domain emotional dictionary L new In the case of , a zero vector is used instead;

[0038] The TF-IDF value of the keyword is used as a weight to multiply the VAD emotional state vector or the zero vector to obtain the corresponding preference knowledge embedding vector;

[0039] Each keyword and its corresponding preference knowledge embedding vector are summarized to construct the user personalized sentiment dictionary.

[0040] Preferably, cosine similarity is used to calculate the similarity between each original word in the text modality information and the keywords in the pre-built user personalized sentiment dictionary.

[0041] Preferably, the method uses the knowledge-enhanced conversation text embedding vector as the main modality, fuses it with the conversation visual embedding vector and the conversation audio embedding vector using a cross-modal Transformer, and obtains the knowledge-enhanced conversation feature vector of each multimodal conversation data; including:

[0042] Using a connection-based temporal classification model, adjusting the sequence lengths of the knowledge-enhanced conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector to a uniform standard;

[0043] Taking the knowledge-enhanced conversation text embedding vector as the primary modality, similarity matrices between the normalized text and audio, and between the text and visual information are calculated by performing a dot product operation on the normalized vectors. The similarity matrices are activated by a Softmax function and used as weight matrices. The similarity matrices are multiplied by the embedding vectors of the corresponding non-text modalities to obtain weighted conversation visual embedding vectors and conversation audio embedding vectors, respectively.

[0044] A cross-modal Transformer is used to fuse the knowledge-enhanced conversation text embedding vector, the weighted conversation visual embedding vector, and the conversation audio embedding vector to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data.

[0045] A knowledge-enhanced emotion recognition system for user conversations in product Q&A communities, based on a pre-set model, including:

[0046] An information acquisition module is configured to acquire historical multi-round question-and-answer information from multiple product question-and-answer communities; each of the historical multi-round question-and-answer information includes a user's multimodal conversation data and its emotion tag, and each piece of the multimodal conversation data includes text, visual, and audio modal information;

[0047] A feature representation module is used to obtain a conversation text embedding vector, a conversation visual embedding vector, and a conversation audio embedding vector corresponding to each piece of multimodal conversation data based on the current historical multi-round question and answer information;

[0048] A preference knowledge embedding module is configured to calculate, for any piece of multimodal conversation data, the similarity between each original word in its text modal information and a keyword in a pre-built user-personalized sentiment dictionary, concatenate the embedding representation of each original word with the preference knowledge embedding vector corresponding to the keyword with the highest similarity, and multiply the concatenation result by the corresponding highest similarity as a weight to obtain a knowledge-enhanced conversation text embedding vector;

[0049] a cross-modal fusion module, configured to use the knowledge-enhanced conversation text embedding vector as the main modality and fuse it with the conversation visual embedding vector and the conversation audio embedding vector using a cross-modal Transformer to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data;

[0050] A conversation scenario modeling module is configured to use an LSTM network to model the contextual emotional dependency of users in each piece of the multimodal conversation data based on the conversation feature vector to obtain a contextual situation vector for personalized preference knowledge perception; and to construct a global conversation graph based on the contextual situation vector, and use a GCN network to model the emotional dependency between users in each piece of the multimodal conversation data to obtain a global context vector for storing the global context state;

[0051] An emotion prediction module, which obtains an emotion recognition prediction result for each piece of the multimodal conversation data through a collaborative attention mechanism based on the interaction between the conversation feature vector and the global context vector;

[0052] A model training module is configured to construct a contrastive learning loss based on a dual-layer supervised contrastive learning strategy of an emotional layer and a multimodal layer according to the conversation text embedding vector, the conversation visual embedding vector, and the conversation audio embedding vector corresponding to each piece of the multimodal conversation data; construct a cross-entropy loss according to the emotion label and emotion recognition prediction result of each piece of the multimodal conversation data; and train the model until convergence by minimizing a joint loss function of the contrastive learning loss and the cross-entropy loss.

[0053] The emotion recognition module is used to use the multi-round question and answer information to be recognized as the input of the converged model and obtain the emotion recognition results of each multimodal dialogue data in the multi-round question and answer information.

[0054] Preferably, a dictionary construction module is further included for constructing the user personalized sentiment dictionary, and the construction process includes:

[0055] The manually annotated sentiment dictionary NRC-VAD Chinese version is selected as the basic sentiment dictionary, which is expressed as Where (V i , A i , D i ) is the word wi The features of represent the corresponding VAD emotional state vector, which respectively represent the scores of activation, pleasure and dominance in three dimensions as V i , A i , D i , and the value range of the three is between 0 and 1;

[0056] Crawl the historical Q&A information of all users in the current product Q&A community, build a Q&A corpus for the target domain, remove stop words to obtain a domain dictionary W, and use this dictionary to train a Word2Vec model to map each word in W to a high-dimensional vector containing the word's semantic information. Use cosine similarity to measure the semantic similarity between any two words.

[0057] For words in W, either belonging to the dictionary L0 or not belonging to the basic sentiment dictionary L0, through the semantic similarity between any two words and several similar words, if there is a similar word belonging to the dictionary L0, then the word is regarded as a potential sentiment word and added to the potential sentiment word set P;

[0058] For each emotion word in the set P, define the algorithm f to map its feature representation into a VAD emotion state vector; if the three dimensions of the VAD emotion state vector are all greater than 0, then add the emotion word and the corresponding VAD emotion state vector to the dictionary L0, and record the expanded domain emotion dictionary as L new ;

[0059] Collect the current user's historical question and answer information and treat it as a document. Remove stop words from the document and then perform word segmentation.

[0060] Use TF-IDF method to extract all keywords from preprocessed documents;

[0061] If the keyword is in the expanded domain sentiment dictionary L new Then find the VAD emotional state vector corresponding to the keyword; if the keyword is not in the expanded domain emotional dictionary L new In the case of , a zero vector is used instead;

[0062] The TF-IDF value of the keyword is used as a weight to multiply the VAD emotional state vector or the zero vector to obtain the corresponding preference knowledge embedding vector;

[0063] Each keyword and its corresponding preference knowledge embedding vector are summarized to construct the user personalized sentiment dictionary.

[0064] A storage medium stores a computer program for identifying emotions in user conversations in a product question-and-answer community for knowledge enhancement, wherein the computer program enables a computer to execute the above-mentioned method for identifying emotions in user conversations in a product question-and-answer community.

[0065] An electronic device, comprising:

[0066] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the product question and answer community user conversation emotion recognition method as described above.

[0067] (3) Beneficial effects

[0068] The present invention provides a method, system, storage medium, and electronic device for identifying emotions in user conversations in a knowledge-enhanced product Q&A community. Compared with existing technologies, this method has the following advantages:

[0069] In this paper, a personalized user sentiment dictionary is constructed through user keyword extraction and dictionary expansion algorithms to obtain a preference knowledge embedding vector. A preference knowledge-enhanced cross-modal Transformer is used to fuse the sentiment knowledge-enhanced conversation text embedding vector, the conversation visual embedding vector, and the conversation audio embedding vector to obtain a knowledge-enhanced conversation feature vector. A personalized preference-aware conversation context GCN network considers the context of multiple rounds of question and answer information and the contextual relationships between users in the conversation to obtain a personalized preference-aware global context vector. Personalized emotion recognition is performed by considering the interaction between the global context vector and the knowledge-enhanced conversation feature vector. Furthermore, a two-layer supervised comparative learning strategy based on the sentiment layer and the multimodal layer is proposed, which enables the model to learn more discriminative multimodal feature representations, further improving the accuracy and granularity of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 A block diagram of a method for identifying emotions in user conversations in a knowledge-enhanced product Q&A community provided by an embodiment of the present invention Figure 2

[0072] Figure 2The process of a method for identifying user emotions in conversations in a knowledge-enhanced product Q&A community provided by an embodiment of the present invention Figure 2

[0073] Figure 3 A schematic diagram of multimodal feature fusion based on a cross-modal Transformer provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0075] The embodiments of the present application solve the technical problem of ignoring that different users have different perceptions and expressions of emotional information by providing a knowledge-enhanced product question-and-answer community user conversation emotion recognition method, system, storage medium and electronic device, thereby achieving personalized emotion recognition.

[0076] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0077] The present invention proposes a personalized user emotion recognition method for multi-round conversation scenarios in a product Q&A community. The key points involved are as follows:

[0078] First, a personalized sentiment dictionary is constructed based on historical Q&A information from users in the current product Q&A community. The preference knowledge embedding vector corresponding to each keyword is queried, and the knowledge-enhanced conversation text embedding vector is fused with the conversation visual embedding vector and the conversation audio embedding vector to obtain the knowledge-enhanced conversation feature vector for each multimodal conversation data.

[0079] Second, through the personalized preference-aware dialogue context GCN network, the context in multi-round question and answer information and the situational relationship between dialogue users are considered to make full use of information from multiple modalities.

[0080] Third, a dual-layer supervised contrastive learning strategy of the sentiment layer and the multimodal layer is proposed, which fully utilizes the consistent relationship between different modal data in the multimodal layer and the consistent relationship between modalities and emotion labels in the low-dimensional VAD sentiment space.

[0081] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0082] Example 1:

[0083] like Figure 1 As shown, an embodiment of the present invention provides a method for recognizing emotions in user conversations in a knowledge-enhanced product question-and-answer community, based on a preset model, including:

[0084] S1. Obtain historical multi-round question-and-answer information from multiple product question-and-answer communities; each of the historical multi-round question-and-answer information includes a user's multimodal conversation data and its emotion label, and each piece of the multimodal conversation data includes text, visual, and audio modal information;

[0085] S2. For the current multi-round question-and-answer information, obtain the conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector corresponding to each piece of the multimodal conversation data;

[0086] S3. For any piece of multimodal conversation data, calculate the similarity between each original word in its text modal information and a keyword in a pre-built user-personalized sentiment dictionary, concatenate the embedding representation of each original word with the preference knowledge embedding vector corresponding to the keyword with the highest similarity, and multiply the concatenation result by the corresponding highest similarity as a weight to obtain a knowledge-enhanced conversation text embedding vector;

[0087] S4. Using the knowledge-enhanced conversation text embedding vector as the main modality, a cross-modal Transformer is used to fuse it with the conversation visual embedding vector and the conversation audio embedding vector to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data.

[0088] S5. Based on the conversation feature vector, using an LSTM network to model the contextual emotional dependency of users in each piece of the multimodal conversation data to obtain a contextual situation vector for personalized preference knowledge perception; and based on the contextual situation vector, constructing a global conversation graph, using a GCN network to model the emotional dependency between users in each piece of the multimodal conversation data to obtain a global context vector for storing a global context state;

[0089] S6. Obtaining an emotion recognition prediction result for each piece of the multimodal conversation data through a collaborative attention mechanism based on the interaction between the conversation feature vector and the global context vector;

[0090] S7. Constructing a contrastive learning loss based on a dual-layer supervised contrastive learning strategy of the sentiment layer and the multimodal layer according to the conversation text embedding vector, the conversation visual embedding vector, and the conversation audio embedding vector corresponding to each piece of the multimodal conversation data; constructing a cross-entropy loss according to the emotion label and emotion recognition prediction result of each piece of the multimodal conversation data; minimizing a joint loss function of the contrastive learning loss and the cross-entropy loss, and training the model until convergence;

[0091] S8. Use the multi-round question and answer information to be recognized as the input of the converged model to obtain the emotion recognition result of each multimodal dialogue data in the multi-round question and answer information.

[0092] In an embodiment of the present invention, a user personalized sentiment dictionary is constructed through user keyword extraction and dictionary expansion algorithms to obtain a preference knowledge embedding vector; a preference knowledge-enhanced cross-modal Transformer is used to fuse the sentiment knowledge-enhanced dialogue text embedding vector, dialogue visual embedding vector, and dialogue audio embedding vector to obtain a knowledge-enhanced dialogue feature vector; a personalized preference-aware dialogue context GCN network is used to consider the context in multiple rounds of question and answer information and the contextual relationship between dialogue users to obtain a personalized preference-aware global context vector; and personalized emotion recognition is performed by considering the interaction between the global context vector and the knowledge-enhanced dialogue feature vector.

[0093] In addition, a two-layer supervised comparative learning strategy based on the sentiment layer and the multimodal layer is proposed to enable the model to learn more discriminative multimodal feature representations, further improving the accuracy and granularity of emotion recognition.

[0094] Figure 2 A flowchart of the emotion recognition method for user conversations in a knowledge-enhanced product question-and-answer community is given. The basic processing flow includes information acquisition, multimodal feature representation, cross-modal fusion with personalized preference knowledge enhancement, conversation scenario modeling with personalized preference knowledge enhancement, emotion recognition prediction, and construction of a user personalized sentiment dictionary. The input is multimodal conversation data, and the output is personalized emotion recognition results.

[0095] Next, we will combine Figure 2 The following steps are described in detail:

[0096] In step S1, historical multi-round question-and-answer information of multiple product question-and-answer communities is obtained; each of the historical multi-round question-and-answer information includes the user's multimodal conversation data and its emotion label, and each of the multimodal conversation data includes text, visual and audio modal information.

[0097] Specifically, any historical multi-round question-answering information obtained is represented as Where |D| is the number of multimodal dialogue data, m = {t, v, a} is the text, visual and audio modality information in the question and answer, is the i-th multimodal dialogue data, sp i is the dialogue user of the i-th multimodal dialogue data, and sp i ∈{S q , S a}, S q and Sa Represents the questioner and the answerer respectively.

[0098] In step S2, for the current historical multi-round question and answer information, the conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector corresponding to each piece of the multimodal conversation data are obtained.

[0099] Exemplary:

[0100] (1) For text modal information, the pre-trained BERT model is used to obtain text feature representation. Input to the BERT model and obtain the conversation text embedding vector by encoding Where R represents is a set of n×dim dimensions.

[0101] Among them, [cls] and [sep] represent the start and end tags of the sentence respectively, n is the length of the text, and dim represents the dimension of the feature vector.

[0102] (2) For visual modality information, in order to effectively utilize the dynamic information in the video, 3D-CNN (3D convolutional neural networks) is used to extract visual features from the video. The 3D filter size is set to 5×5×5, and the dialogue visual embedding vector corresponding to the i-th multimodal dialogue data is obtained.

[0103] (3) For audio modal information, the open source speech processing toolkit openSMILE is used to extract audio features. The configuration file uses IS13ComParE to extract features such as fundamental frequency, pitch, Mel-frequency cepstral coefficients (MFSDs), and Mel-frequency spectrum energy (MFBs) at a frame rate of 30 Hz and a sliding window of 100 ms. These acoustic features cover different types such as sound quality, rhythm, and spectrum and are related to intonation and emotion. The conversation audio embedding vector corresponding to the i-th multimodal conversation data is obtained.

[0104] In step S3, for any piece of the multimodal conversation data, the similarity between each original word in its text modal information and the keywords in the pre-built user personalized sentiment dictionary is calculated, the embedding representation of each original word is concatenated with the preference knowledge embedding vector corresponding to the keyword with the highest similarity, and the corresponding highest similarity is multiplied with the concatenation result as a weight to obtain a knowledge-enhanced conversation text embedding vector.

[0105] To address the problem that existing emotion recognition methods ignore individual users' personalized preferences for emotional vocabulary, we first construct a personalized emotion lexicon for each user. Traditional discrete models of emotion quantification assume that the emotional space consists of discrete and finite basic emotions, ignoring the correlation between emotions. Therefore, we introduce the NRC-VAD Chinese version of the emotion lexicon, based on the VAD psychological emotion model, as a basic lexicon. This lexicon defines the emotional state of a word using numerical values ​​along three dimensions: valence (emotional polarity), arousal (calmness or excitement), and dominance (perceived control). The distance between pairs of words can be calculated in a continuous emotion space, thereby capturing the potential relationships between the emotions implied by each word.

[0106] In addition, in order to accurately capture the preferences of individual users in questions and answers, an embodiment of the present invention adopts the TF-IDF method to analyze the distribution of keywords in the user's historical question and answer data. By calculating the TF-IDF value of each word, it is possible to quantify the frequency with which users use specific words in their personal historical data and evaluate the distinctiveness of these words in the entire user group. Words with higher TF-IDF values ​​indicate that they not only appear frequently in the historical questions and answers of a single user, but also have a lower prevalence in the overall question and answer dataset, thereby revealing the user's significant preference for these words when publishing content.

[0107] like Figure 2 As shown, the construction of the user personalized sentiment dictionary is as follows:

[0108] (1) Constructing an expanded domain sentiment dictionary

[0109] Words in text modalities contain semantic emotional information, but existing word-level text feature extraction methods have difficulty learning implicit emotional state information. Therefore, introducing an external sentiment dictionary as a priori knowledge into the model can supplement the emotional information of words. To obtain the emotional state information corresponding to the word-level text feature representation in question and answer, the embodiment of the present invention designs a domain emotion dictionary expansion method. The specific steps are as follows:

[0110] Step 1: Select the manually annotated sentiment dictionary NRC-VAD Chinese version as the basic sentiment dictionary, expressed as The dictionary contains VAD scores of 20,000 words; i , A i , D i ) is the word w i The features of represent the corresponding VAD emotional state vector, which respectively represent the scores of activation, pleasure and dominance in three dimensions as V i , A i , D i, and the value range of the three is between 0 and 1.

[0111] Note that the vocabulary in L0 comes from commonly used sentiment lexicons and Twitter corpus.

[0112] Step 2: Crawl historical Q&A information from users in the current product Q&A community, build a Q&A corpus for the target domain, remove stop words to obtain a domain dictionary W, and use this dictionary to train a Word2Vec model to map each word in W to a high-dimensional vector containing the word's semantic information. Use cosine similarity to measure the semantic similarity between any two words.

[0113] Step 3: For words in W, either belonging to the dictionary L0 or not belonging to the basic sentiment dictionary L0, the semantic similarity between any two words is used to find several similar words. If there is a similar word belonging to the dictionary L0, then the word is regarded as a potential sentiment word and added to the potential sentiment word set P;

[0114] Step 4: For each emotion word in the set P, define the algorithm f to map its feature representation into a VAD emotion state vector; if the three dimensions of the VAD emotion state vector are all greater than 0, then add the emotion word and the corresponding VAD emotion state vector to the dictionary L0, and record the expanded domain emotion dictionary as L new .

[0115] (2) Build a user-personalized sentiment dictionary

[0116] Collecting historical question-and-answer information of users in the multimodal conversation data and treating it as a document, removing stop words from the document and performing word segmentation processing;

[0117] The TF-IDF method is used to extract all keywords from the preprocessed documents. It includes two parts: TF and IDF. TF stands for term frequency, which is the frequency of a word appearing in the document. IDF stands for inverse document frequency, which is used to measure the general importance of a word. The TF-IDF value of a word is obtained by multiplying the TF and IDF parts. The higher the value, the more important the word is in the document. The calculation process is shown in formula (1):

[0118]

[0119] in, represents the frequency of word i in the historical questions and answers of user spi, |U| represents the number of historical questions and answers in the user's historical question and answer information set, which is equivalent to the number of users; k k,sp i Indicates user sp i The number of all words in the history quiz, is the number of users whose historical question and answer information contains word t.

[0120] The keywords extracted by the TF-IDF method reflect the user's personalized vocabulary preferences. In order to extract the user's keywords, a threshold is set and words with TF-IDF values ​​higher than the threshold are regarded as the user's keywords.

[0121] Then determine if the keyword is in the expanded domain sentiment dictionary L new Then find the VAD emotional state vector corresponding to the keyword; if the keyword is not in the expanded domain emotional dictionary L new In the case of , a zero vector is used instead;

[0122] For the multimodal conversation data in the training phase, if a keyword in the user's historical information appears in the conversation text of the questioner or the answerer, the TF-IDF value of the keyword is used as a weight and multiplied by the VAD emotional state vector to obtain a user personalized emotional knowledge embedding vector representing the user's preference knowledge.

[0123] Finally, each keyword and the corresponding preference knowledge embedding vector are summarized to construct the user personalized sentiment dictionary.

[0124] In order to integrate the user's personalized preference information into the multimodal feature representation, the user's personalized sentiment dictionary is used to obtain the preference knowledge embedding vector corresponding to each original word in the text modal information.

[0125] For any of the multimodal conversation data, Figure 3 As shown, this step specifically calculates each original word w in the text modal information i The keyword l in the constructed user personalized sentiment dictionary i The cosine similarity sim(w i , l i ); For the original word w in the text modal information i , select the dictionary with w i Similarity sim(w i , l i )Highest keyword l i , the keyword l i The corresponding personalized sentiment preference knowledge embedding vector and the original word w i After the word embedding is concatenated, w i With l i The similarity is used as a weight to multiply the concatenated vector to obtain the knowledge-enhanced text embedded representation.

[0126] In step S4, the knowledge-enhanced conversation text embedding vector is used as the main modality and is fused with the conversation visual embedding vector and the conversation audio embedding vector using a cross-modal Transformer to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data.

[0127] As a basic tool for human communication, language plays a vital role in the field of emotion perception. It can not only directly convey information, but also implicitly express emotions and attitudes. In the study of sentiment analysis, the introduction of metadata about the emotional state of words in the VAD (Valence-Arousal-Dominance) sentiment dictionary can significantly enhance the richness and accuracy of emotional information in text data. Therefore, the embodiment of the present invention regards text as the dominant modality, and through its rich semantic and emotional content, guides the deep integration of other modal data (such as audio, video, etc.) with text.

[0128] Specifically, this step uses the knowledge-enhanced conversation text embedding vector as the query vector to perform cross-modal attention interaction with the conversation visual embedding vector and the conversation audio embedding vector, learns the audio and visual features of preference knowledge perception, and filters out emotion-irrelevant noise information in non-language modalities; the multimodal features are jointly input into the cross-modal Transformer layer to obtain a multimodal feature representation enhanced with preference knowledge.

[0129] The fusion process is as follows Figure 3 The steps are as follows:

[0130] Step 1: First, due to the different sampling rates of data in different modalities, the sequence lengths of different modalities will vary. In order to perform subsequent calculations, the multimodal data must first be aligned. Specifically, the Connectionist Temporal Classification (CTC) model is used to adjust the sequence lengths of the knowledge-enhanced conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector to a unified standard. As a sequence-to-sequence mapping method, the CTC model can effectively process sequence data of different lengths and provides an effective solution for multimodal data alignment. A linear network is then used to unify the vector dimensions of data in different modalities. The specific calculation is shown in formula (2):

[0131]

[0132] Among them, T i , V i , A i ∈R L×d , T iis the standardized preference knowledge-aware text feature representation corresponding to the i-th multimodal dialogue data, V i 、A i is the standardized visual and audio feature vector corresponding to the i-th multimodal conversation data, L is the sequence length, d represents the feature dimension; MLP is a multi-layer perceptron.

[0133] The knowledge-enhanced conversation text embedding vector is obtained by concatenating the user's personalized emotion knowledge embedding vector with the corresponding conversation text embedding vector. Through this transformation, text, visual, and audio features can be mapped into a common high-dimensional space.

[0134] Step 2: Calculate the similarity matrix between the normalized text and audio, and between the text and visual information through the normalized vector dot product operation. Then, activate the similarity matrix with the Softmax function as a weight matrix and multiply it with the embedding vector of the corresponding non-text modality to obtain the weighted dialogue visual embedding vector and dialogue audio embedding vector respectively; reduce the difference between the non-language modality and the text modality. The specific process is shown in formulas (3) to (6):

[0135]

[0136] in, represents the similarity matrix, and is the weighted visual and audio feature representation.

[0137] Step 3: Use a cross-modal Transformer layer to fuse the knowledge-enhanced conversation text embedding vector, the weighted conversation visual embedding vector, and the conversation audio embedding vector to obtain a knowledge-enhanced conversation feature vector.

[0138] The core of the cross-modal Transformer is the cross-modal attention mechanism. We use the text feature T i As the query vector, As the key vector, The value vector is input into the cross-modal attention. The specific process is shown in formula (7):

[0139]

[0140] Among them, Q, K, and V are the inputs of the three modalities after linear mapping, and dim is the feature dimension. The vector F after multimodal interaction obtained by the attention layer is f The final dialogue feature vector is generated after being processed by the residual connection and normalization layer, the feedforward layer, the residual connection and normalization layer. The specific process is shown in formulas (8) to (9):

[0141]

[0142] Among them, f θ represents the feedforward network, Norm represents normalization, F i is the final output of the cross-modal Transformer, i.e., the knowledge-enhanced dialogue feature vector corresponding to the i-th multimodal dialogue data.

[0143] It can be seen that the embodiment of the present invention takes into account the user's personal emotional preference information, can more accurately identify the user's emotions, and provides an effective solution for personalized emotion recognition.

[0144] In step S5, based on the conversation feature vector, an LSTM network is used to model the contextual emotional dependency of users in each piece of the multimodal conversation data to obtain a contextual situation vector of personalized preference knowledge perception; and based on the contextual situation vector, a global conversation graph is constructed, and a GCN network is used to model the emotional dependency between users in each piece of the multimodal conversation data to obtain a global situation vector that stores the global context state.

[0145] In order to model the influence of the emotional state of the user's historical speech on the current speech emotion, the LSTM network is used to capture the user's emotional state. The gating mechanism (including the forget gate and the input gate) is used to determine which information should be retained and which should be forgotten, so as to capture the long-distance dependency in the time series data. In the LSTM module of each user P, the input is the conversation feature vector The output is taking into account the corpus Contextual Situation Vector for Personalized Preference Knowledge Awareness

[0146] Next, we construct a global conversation graph, where the nodes are the output of LSTM. Initialization assignment, the node set can be represented as shown Where P∈{S q , S a}, N d is the total number of sentences in the conversation d. The nodes are connected strictly in the chronological order of the conversation. By using GCN to update the graph nodes, the emotional dependency relationship between users in each multimodal conversation data is modeled to obtain a global context vector that stores the global context state, expressed as

[0147] In step S6, based on the interaction between the dialogue feature vector and the global context vector, an emotion recognition prediction result of each piece of the multimodal dialogue data is obtained through a collaborative attention mechanism.

[0148] When classifying the emotion of the i-th multimodal conversation data, its knowledge-enhanced conversation feature vector can be interacted with M based on the attention mechanism, thereby supplementing the single conversation data with rich context information. The specific calculation is shown in formulas (10) to (12):

[0149] att=softmax((M) T u i ) (10)

[0150]

[0151] Among them, att is the learned attention weight vector, which is calculated by weighted summation of each multimodal dialogue data in the memory bank using att It contains both the summary of the contextual emotional state and the information of the conversation data itself, and finally the weighted memory vector For the i-th multimodal dialogue data Update the feature representation.

[0152] Then the updated speech vector It is converted into emotion classification probability distribution through linear transformation and softmax function, as shown in formula (13):

[0153]

[0154] Among them, the output vector y ij is the probability value of the i-th multimodal dialogue data being identified as the j-th emotion, and its dimension is equal to the number of emotion categories, W oj 、b j The weight parameters and bias terms in the linear transformation layer corresponding to the j-th emotion.

[0155] In step S7, a contrastive learning loss is constructed based on a two-layer supervised contrastive learning strategy of the emotion layer and the multimodal layer according to the conversation text embedding vector, the conversation visual embedding vector, and the conversation audio embedding vector corresponding to each piece of the multimodal conversation data; a cross-entropy loss is constructed according to the emotion label and the emotion recognition prediction result of each piece of the multimodal conversation data; and the joint loss function of the contrastive learning loss and the cross-entropy loss is minimized, and the model is trained until convergence.

[0156] In the related art, the multimodal emotion recognition method does not take into account the semantic similarity relationship between emotions. For example, "happy" is more similar to "excited", but has a greater semantic difference from "sad". In particular, when the sample distribution of different emotion categories is uneven, it is difficult to distinguish emotions with similar semantics. The embodiment of the present invention proposes a two-layer supervised contrastive learning strategy on the emotional layer and the multimodal layer, which makes full use of the consistency relationship between the multimodal data and the emotion label in a single sample instance, as well as the consistency relationship between the multimodal data of different sample instances, and constructs a contrastive learning loss function, so that even in the case of unbalanced samples, more discriminative multimodal feature representations can be learned, which is conducive to distinguishing semantically similar emotions and improving the computational efficiency of contrastive learning. The details are as follows:

[0157] (1) Obtaining contrastive learning loss based on sentiment layer

[0158] Traditional supervised contrastive learning methods, when processing emotion labels, often simply treat samples with different emotion labels as negative examples. This approach ignores the quantitative semantic similarity between emotion labels, and emotions are continuous rather than discrete in semantic space. Due to the complexity and diversity of emotions, simply defining positive and negative examples based on the similarities and differences in emotion labels is insufficient to capture the nuances of emotions. This approach can cause the model to have difficulty distinguishing semantically similar emotions, thus affecting the accuracy and robustness of emotion recognition.

[0159] In order to solve the above problem, the conversation text embedding vector, conversation visual embedding vector and conversation audio embedding vector are first mapped to the three-dimensional VAD (Valence-Arousal-Dominance) space. The specific process is shown in formula (14).

[0160] Map the conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector into the three-dimensional VAD emotion space:

[0161]

[0162] Among them, the modality m∈{T,V,A}, T,V,A are the text, visual and audio modal information in the multimodal conversation data respectively; is the corresponding multimodal embedding vector, is the feature representation mapped to the VAD emotion space, i is the index of the multimodal dialogue data; W is the weight parameter; and b is the bias term.

[0163] Then, manually labeled e is introduced for each emotion in the VAD emotion space. c ∈R 3 ; where c∈E represents the emotion category, E represents the set of emotion label categories, e cRepresents the VAD vector corresponding to the true emotion label;

[0164] Since the feature representation vector of each modality in the VAD emotion space should be close to the VAD vector corresponding to its true emotion label and far away from the VAD vectors of other emotion labels, each multimodal conversation data is taken as a sample, and the feature representation of the three modal information in each sample mapped to the VAD emotion space and the VAD vector corresponding to its true emotion label are regarded as positive sample pairs, and the VAD vectors of other types of emotions are regarded as negative sample pairs. The first contrastive learning loss is constructed as shown in formula (15):

[0165]

[0166] Where |E| is the number of emotion categories; due to the different quality and importance of different modalities, the hyperparameter λ m Used to adjust the weight of the loss function for the consistency comparison between each modality information and the label; log is the logarithmic function; Represents the true emotion label y of the i-th multimodal dialogue data i The corresponding VAD vector; represents the dot product operation, which is used to calculate the similarity between vectors; τ is a learnable hyperparameter.

[0167] (2) The process of obtaining contrastive learning loss based on the multimodal layer; including:

[0168] The similarity between the multimodal feature representations of sample instances belonging to different emotion categories should be lower, and the similarity between the multimodal data corresponding to the same emotion samples should be higher. Therefore, the multimodal embedding vectors of samples with the same emotion are regarded as positive sample pairs, and the multimodal embedding vectors of samples with different emotion categories are regarded as negative sample pairs. The second contrastive learning loss is constructed as shown in formula (16):

[0169]

[0170] Among them, m1 and m2 are two arbitrary modes; λ m1，m2 is the weight hyperparameter for contrastive learning between different modalities; Nd is the number of samples of previous multi-round question-answering information; The label is y in the current historical multi-round question and answer information i The number of samples; j, k are the indexes of the multimodal conversation data used as samples.

[0171] From this, the total loss function of the two-layer supervised contrastive learning strategy based on the sentiment layer and the multimodal layer can be determined As shown in formula (17):

[0172]

[0173] In addition, this step also constructs the cross entropy loss based on the real emotion label The weighted summation with the contrastive learning loss shown in formula (14) is performed to construct the joint loss function As shown in formulas (18) to (19):

[0174]

[0175] The model is trained until convergence by minimizing the joint loss function of the contrastive learning loss and the cross entropy loss.

[0176] In step S8, the multi-round question-and-answer information to be recognized is used as the input of the converged model to obtain the emotion recognition result of each multimodal dialogue data in the multi-round question-and-answer information.

[0177] Thus, the embodiment of the present invention has completed the personalized emotion recognition of the user based on the historical multi-round question and answer information of the product question and answer community.

[0178] In order to demonstrate the advantages of the embodiments of the present invention, the following examples are provided:

[0179] This example covers the entire process from data acquisition to model training and testing. It provides a method for recognizing emotions in conversations within a product Q&A community based on sentiment preference awareness. This method can effectively improve the performance of recognizing emotions among commenters, helping companies better understand user attitudes toward their products and thus improve their products and services.

[0180] (1) Using Python web crawler technology, we crawled product-related question and answer corpus on the product question and answer community. The main crawled content includes the text, video and audio content in the questions and answers, as well as the user IDs of the questioner and the answerer, to form a multimodal question and answer dataset, in which the audio data is extracted from the video file.

[0181] (2) In the multimodal feature extraction layer, the question-answer text, video, and audio data are encoded using pre-trained feature extractors to obtain the corresponding text, video, and audio feature representations.

[0182] (3) Fuse the different modal information extracted in step (2).

[0183] ① Crawl user-generated content (Q&A and comment text) related to the domain to build a domain corpus. Use this domain corpus to expand the basic sentiment lexicon to obtain the sentiment lexicon for the current Q&A domain. Crawl historical user answers and use the TF-IDF method to extract each user's keywords. Use the TF-IDF value as a weight and multiply it with the VAD vector to construct the user's personalized sentiment preference knowledge, thereby building a personalized sentiment lexicon.

[0184] ② For any of the multimodal conversation data, calculate the similarity between each word in the text modal information and the words in the constructed user personalized sentiment dictionary; for the original word w in the text modal information, select the keyword l with the highest similarity to w in the user personalized sentiment dictionary, concatenate the preference knowledge embedding vector corresponding to the keyword, and multiply the concatenated vector by the similarity between w and, as a weight, to obtain a knowledge-enhanced conversation text embedding vector.

[0185] (4) The information of different modalities obtained in step (2) is used to construct a contrast loss function based on the two-layer supervised contrast learning strategy of the sentiment layer and the multimodal layer.

[0186] (5) The multimodal fusion features of each speech segment of the same user in the question-answering dialogue are input into the long short-term memory network (LSTM). The LSTM network is used to model the emotional influence relationship between contexts and capture the dynamic changes of emotional states over time.

[0187] (6) The dialogue information of the questioner and the answerer is integrated, and a dialogue graph is constructed according to the time sequence of the dialogue. The GCN network is used to update the graph nodes, thereby modeling the persistent emotional influence between speakers in the global dialogue. The emotional state vectors obtained at each time step are spliced ​​to obtain a global dialogue context vector.

[0188] (7) In the prediction module, the multimodal feature representation of the sentence to be predicted is input into the classifier after attention interaction with the global dialogue context vector to obtain the prediction result of the emotion category.

[0189] (8) Construct a cross entropy loss function based on the real emotion label, perform weighted summation with the double-layer contrastive learning loss obtained in step (4), construct a total loss function, and train the model until convergence.

[0190] (9) The performance of the model is evaluated on the test set using four classification indicators, namely accuracy, weighted average F1 value, weighted average recall rate and weighted average precision rate. The model parameters are tuned according to these indicator values, and the tuned model is trained and adjusted continuously until the performance of the model reaches the optimal level.

[0191] Example 2:

[0192] An embodiment of the present invention provides a knowledge-enhanced product question-and-answer community user conversation emotion recognition system, based on a preset model, including:

[0193] An information acquisition module is configured to acquire historical multi-round question-and-answer information from multiple product question-and-answer communities; each of the historical multi-round question-and-answer information includes a user's multimodal conversation data and its emotion tag, and each piece of the multimodal conversation data includes text, visual, and audio modal information;

[0194] A feature representation module is used to obtain a conversation text embedding vector, a conversation visual embedding vector, and a conversation audio embedding vector corresponding to each piece of multimodal conversation data based on the current historical multi-round question and answer information;

[0195] A preference knowledge embedding module is configured to calculate, for any piece of multimodal conversation data, the similarity between each original word in its text modal information and a keyword in a pre-built user-personalized sentiment dictionary, concatenate the embedding representation of each original word with the preference knowledge embedding vector corresponding to the keyword with the highest similarity, and multiply the concatenation result by the corresponding highest similarity as a weight to obtain a knowledge-enhanced conversation text embedding vector;

[0196] a cross-modal fusion module, configured to use the knowledge-enhanced conversation text embedding vector as the main modality and fuse it with the conversation visual embedding vector and the conversation audio embedding vector using a cross-modal Transformer to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data;

[0197] A conversation scenario modeling module is configured to use an LSTM network to model the contextual emotional dependency of users in each piece of the multimodal conversation data based on the conversation feature vector to obtain a contextual situation vector for personalized preference knowledge perception; and to construct a global conversation graph based on the contextual situation vector, and use a GCN network to model the emotional dependency between users in each piece of the multimodal conversation data to obtain a global context vector for storing the global context state;

[0198] An emotion prediction module, which obtains an emotion recognition prediction result for each piece of the multimodal conversation data through a collaborative attention mechanism based on the interaction between the conversation feature vector and the global context vector;

[0199] A model training module is configured to construct a contrastive learning loss based on a dual-layer supervised contrastive learning strategy of an emotional layer and a multimodal layer according to the conversation text embedding vector, the conversation visual embedding vector, and the conversation audio embedding vector corresponding to each piece of the multimodal conversation data; construct a cross-entropy loss based on the emotion label and emotion recognition prediction result of each piece of the multimodal conversation data; and train the model until convergence by minimizing a joint loss function of the contrastive learning loss and the cross-entropy loss.

[0200] The emotion recognition module is used to use the multi-round question and answer information to be recognized as the input of the converged model and obtain the emotion recognition results of each multimodal dialogue data in the multi-round question and answer information.

[0201] In an optional embodiment, the knowledge-enhanced product question-and-answer community user dialogue emotion recognition system further includes a dictionary construction module for constructing the user personalized emotion dictionary. The construction process includes two steps:

[0202] The manually annotated sentiment dictionary NRC-VAD Chinese version is selected as the basic sentiment dictionary, which is expressed as Where (V i , A i , D i ) is the word w i The features of represent the corresponding VAD emotional state vector, which respectively represent the scores of activation, pleasure and dominance in three dimensions as V i , A i , D i , and the value range of the three is between 0 and 1;

[0203] Crawl the historical Q&A information of all users in the current product Q&A community, build a Q&A corpus for the target domain, remove stop words to obtain a domain dictionary W, and use this dictionary to train a Word2Vec model to map each word in W to a high-dimensional vector containing the word's semantic information. Use cosine similarity to measure the semantic similarity between any two words.

[0204] For words in W, either belonging to the dictionary L0 or not belonging to the basic sentiment dictionary L0, through the semantic similarity between any two words and several similar words, if there is a similar word belonging to the dictionary L0, then the word is regarded as a potential sentiment word and added to the potential sentiment word set P;

[0205] For each emotion word in the set P, define the algorithm f to map its feature representation into a VAD emotion state vector; if the three dimensions of the VAD emotion state vector are all greater than 0, then add the emotion word and the corresponding VAD emotion state vector to the dictionary L0, and record the expanded domain emotion dictionary as L new ;

[0206] Collect the current user's historical question and answer information and treat it as a document. Remove stop words from the document and then perform word segmentation.

[0207] Use TF-IDF method to extract all keywords from preprocessed documents;

[0208] If the keyword is in the expanded domain sentiment dictionary L newThen find the VAD emotional state vector corresponding to the keyword; if the keyword is not in the expanded domain emotional dictionary L new In the case of , a zero vector is used instead;

[0209] The TF-IDF value of the keyword is used as a weight to multiply the VAD emotional state vector or the zero vector to obtain the corresponding preference knowledge embedding vector;

[0210] Each keyword and its corresponding preference knowledge embedding vector are summarized to construct the user personalized sentiment dictionary.

[0211] Example 3:

[0212] An embodiment of the present invention provides a storage medium storing a computer program for identifying emotions in user conversations in a product question-and-answer community for knowledge enhancement, wherein the computer program enables a computer to execute the method for identifying emotions in user conversations in a product question-and-answer community as described in Example 1.

[0213] Example 4:

[0214] An embodiment of the present invention provides an electronic device, including:

[0215] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the product question and answer community user conversation emotion recognition method as described in Example 1.

[0216] It can be understood that the knowledge-enhanced product Q&A community user conversation emotion recognition system, storage medium and electronic device provided in the embodiments of the present invention correspond to the knowledge-enhanced product Q&A community user conversation emotion recognition method provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the product Q&A community user conversation emotion recognition method, and will not be repeated here.

[0217] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0218] 1. The embodiment of the present invention comprehensively considers the influence relationship between the contexts in multiple rounds of question and answer information and the emotional state of the dialogue users, and fully utilizes the multimodal dialogue data on the product question and answer platform to improve the emotion recognition effect, help the platform and enterprises understand the emotional state of users, and thus improve user satisfaction.

[0219] 2. This embodiment of the present invention constructs a user-personalized sentiment dictionary based on a question-and-answer corpus and historical user question-and-answer information to supplement the implicit sentiment information in the text. By calculating the similarity between the original words in the text modal information and the keywords in the user's personalized sentiment dictionary, the embedding representation of each original word is concatenated with the preference knowledge embedding vector corresponding to the keyword with the highest similarity. The corresponding highest similarity is used as a weight and multiplied with the concatenated result to obtain a knowledge-enhanced conversation text embedding vector. This vector is then used as the main modality to filter out sentiment-irrelevant noise information in non-text modalities. The cross-modal Transformer is then used to learn information interaction between multiple modalities. This module can reduce unnecessary information interference and achieve deep fusion of multimodal data, thereby obtaining higher-quality fusion feature representations.

[0220] 3. This embodiment of the present invention proposes a dual-layer supervised contrastive learning strategy in the sentiment layer and the multimodal layer. This strategy externally introduces label-related emotional knowledge and leverages the similarities between sample modal data of the same type and the differences between sample modal data of different types, as well as the similarities between modalities and their corresponding labels, and the differences between modalities and other labels. In this way, the model can capture the subtle differences between different emotional states and effectively distinguish semantically similar emotion categories. This contrastive learning strategy enables the model to learn more discriminative feature representations, improving the accuracy and granularity of emotion recognition.

[0221] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0222] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A knowledge-enhanced method for identifying user emotions in conversations in a product Q&A community, characterized by: Based on preset models, including: Obtaining historical multi-round question-and-answer information from multiple product question-and-answer communities; each of the historical multi-round question-and-answer information includes a user's multimodal conversation data and its emotion label, and each piece of the multimodal conversation data includes text, visual, and audio modal information; For the current multi-round question-and-answer information, obtain the conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector corresponding to each piece of the multimodal conversation data; For any piece of multimodal conversation data, the similarity between each original word in its text modal information and a keyword in a pre-built user-personalized sentiment dictionary is calculated, the embedding representation of each original word is concatenated with the preference knowledge embedding vector corresponding to the keyword with the highest similarity, and the corresponding highest similarity is used as a weight to multiply the concatenated result to obtain a knowledge-enhanced conversation text embedding vector; Taking the knowledge-enhanced conversation text embedding vector as the main modality, and fusing it with the conversation visual embedding vector and the conversation audio embedding vector using a cross-modal Transformer to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data; Based on the conversation feature vector, an LSTM network is used to model the contextual emotional dependency of users in each piece of the multimodal conversation data to obtain a contextual situation vector for personalized preference knowledge perception; and based on the contextual situation vector, a global conversation graph is constructed, and a GCN network is used to model the emotional dependency between users in each piece of the multimodal conversation data to obtain a global situation vector that stores a global context state. Obtaining an emotion recognition prediction result for each piece of the multimodal conversation data through a collaborative attention mechanism based on the interaction between the conversation feature vector and the global context vector; Constructing a contrastive learning loss based on a dual-layer supervised contrastive learning strategy of the sentiment layer and the multimodal layer according to the conversation text embedding vector, the conversation visual embedding vector, and the conversation audio embedding vector corresponding to each piece of the multimodal conversation data; constructing a cross-entropy loss based on the emotion label and emotion recognition prediction result of each piece of the multimodal conversation data; minimizing a joint loss function of the contrastive learning loss and the cross-entropy loss, and training the model until convergence; The multi-round question-and-answer information to be recognized is used as the input of the converged model to obtain the emotion recognition results of each multimodal dialogue data in the multi-round question-and-answer information.

2. The method for identifying emotions in user conversations in a product question-and-answer community according to claim 1, wherein: The process of obtaining contrastive learning loss based on sentiment layer; include Map the conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector into the three-dimensional VAD emotion space: Among them, the modality m∈{T,V,A}, T,V,A are the text, visual and audio modal information in the multimodal conversation data respectively; is the corresponding multimodal embedding vector, is the feature representation mapped to the VAD emotion space, i is the index of the multimodal conversation data; R is the set; W is the weight parameter; b is the bias term; Introducing manually labeled e for each emotion in the VAD emotion space c ∈R 3 ; where c∈E represents the emotion category, E represents the set of emotion label categories, e c Represents the VAD vector corresponding to the true emotion label; Each piece of multimodal conversation data is taken as a sample. The feature representation of the three modal information in each sample is mapped to the VAD emotion space. The VAD vector corresponding to its true emotion label is regarded as a positive sample pair, and the VAD vector of other types of emotions is regarded as a negative sample pair. The first contrastive learning loss is constructed: Where |E| is the number of emotion categories; the hyperparameter λ m Used to adjust the weight of the loss function for the consistency comparison between each modality information and the label; log is the logarithmic function; Represents the true emotion label y of the i-th multimodal dialogue data i The corresponding VAD vector; · represents the dot product operation, which is used to calculate the similarity between vectors; τ is a learnable hyperparameter.

3. The method for identifying emotions in user conversations in a product question-and-answer community according to claim 2, wherein: The process of obtaining contrastive learning loss based on multimodal layers; including: The multimodal embedding vectors of samples with the same emotion are regarded as positive sample pairs, and the multimodal embedding vectors of samples with different emotions are regarded as negative sample pairs, and the second contrastive learning loss is constructed: Among them, m1 and m2 are two arbitrary modes; λ m1,m2 is the weight hyperparameter for contrastive learning between different modalities; Nd is the number of samples of previous multi-round question-answering information; The label is y in the current historical multi-round question and answer information i The number of samples; j, k are the indexes of the multimodal conversation data used as samples.

4. The method for identifying emotions in user conversations in a product question-and-answer community according to claim 1, wherein: The process of constructing the user personalized sentiment dictionary includes: The manually annotated sentiment dictionary NRC-VAD Chinese version is selected as the basic sentiment dictionary, which is expressed as Where (V i , A i , D i ) is the word w i The features of represent the corresponding VAD emotional state vector, which respectively represent the scores of activation, pleasure and dominance in three dimensions as V i , A i , D i , and the value range of the three is between 0 and 1; Crawl the historical Q&A information of all users in the current product Q&A community, build a Q&A corpus for the target domain, remove stop words to obtain a domain dictionary W, and use this dictionary to train a Word2Vec model to map each word in W to a high-dimensional vector containing the word's semantic information. Use cosine similarity to measure the semantic similarity between any two words. For words in W, either belonging to the dictionary L0 or not belonging to the basic sentiment dictionary L0, through the semantic similarity between any two words and several similar words, if there is a similar word belonging to the dictionary L0, then the word is regarded as a potential sentiment word and added to the potential sentiment word set P; For each emotion word in the set P, define the algorithm f to map its feature representation into a VAD emotion state vector; if the three dimensions of the VAD emotion state vector are all greater than 0, then add the emotion word and the corresponding VAD emotion state vector to the dictionary L0, and record the expanded domain emotion dictionary as L new ; Collect the current user's historical question and answer information and treat it as a document. Remove stop words from the document and then perform word segmentation. Use TF-IDF method to extract all keywords from preprocessed documents; If the keyword is in the expanded domain sentiment dictionary L new Then find the VAD emotional state vector corresponding to the keyword; if the keyword is not in the expanded domain emotional dictionary L new In the case of , a zero vector is used instead; The TF-IDF value of the keyword is used as a weight to multiply the VAD emotional state vector or the zero vector to obtain the corresponding preference knowledge embedding vector; Each keyword and its corresponding preference knowledge embedding vector are summarized to construct the user personalized sentiment dictionary.

5. The method for identifying emotions in user conversations in a product question-and-answer community according to claim 1, wherein: Cosine similarity is used to calculate the similarity between each original word in the text modality information and the keywords in the pre-built user personalized sentiment dictionary.

6. The method for identifying emotions in user conversations in a product question-and-answer community according to claim 1, wherein: The method uses the knowledge-enhanced conversation text embedding vector as the main modality, fuses it with the conversation visual embedding vector and the conversation audio embedding vector using a cross-modal Transformer, and obtains the knowledge-enhanced conversation feature vector of each multimodal conversation data; including: Using a connection-based temporal classification model, adjusting the sequence lengths of the knowledge-enhanced conversation text embedding vector, conversation visual embedding vector, and conversation audio embedding vector to a uniform standard; Taking the knowledge-enhanced conversation text embedding vector as the primary modality, similarity matrices between the normalized text and audio, and between the text and visual information are calculated by performing a dot product operation on the normalized vectors. The similarity matrices are activated by a Softmax function and used as weight matrices. The similarity matrices are multiplied by the embedding vectors of the corresponding non-text modalities to obtain weighted conversation visual embedding vectors and conversation audio embedding vectors, respectively. A cross-modal Transformer is used to fuse the knowledge-enhanced conversation text embedding vector, the weighted conversation visual embedding vector, and the conversation audio embedding vector to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data.

7. A knowledge-enhanced product question-and-answer community user conversation emotion recognition system, characterized by: Based on preset models, including: An information acquisition module is configured to acquire historical multi-round question-and-answer information from multiple product question-and-answer communities; each of the historical multi-round question-and-answer information includes a user's multimodal conversation data and its emotion tag, and each piece of the multimodal conversation data includes text, visual, and audio modal information; A feature representation module is used to obtain a conversation text embedding vector, a conversation visual embedding vector, and a conversation audio embedding vector corresponding to each piece of multimodal conversation data based on the current historical multi-round question and answer information; A preference knowledge embedding module is configured to calculate, for any piece of multimodal conversation data, the similarity between each original word in its text modal information and a keyword in a pre-built user-personalized sentiment dictionary, concatenate the embedding representation of each original word with the preference knowledge embedding vector corresponding to the keyword with the highest similarity, and multiply the concatenation result by the corresponding highest similarity as a weight to obtain a knowledge-enhanced conversation text embedding vector; a cross-modal fusion module, configured to use the knowledge-enhanced conversation text embedding vector as the main modality and fuse it with the conversation visual embedding vector and the conversation audio embedding vector using a cross-modal Transformer to obtain a knowledge-enhanced conversation feature vector for each of the multimodal conversation data; A conversation scenario modeling module is configured to use an LSTM network to model the contextual emotional dependency of users in each piece of the multimodal conversation data based on the conversation feature vector to obtain a contextual situation vector for personalized preference knowledge perception; and to construct a global conversation graph based on the contextual situation vector, and use a GCN network to model the emotional dependency between users in each piece of the multimodal conversation data to obtain a global context vector for storing the global context state; An emotion prediction module, which obtains an emotion recognition prediction result for each piece of the multimodal conversation data through a collaborative attention mechanism based on the interaction between the conversation feature vector and the global context vector; A model training module is configured to construct a contrastive learning loss based on a dual-layer supervised contrastive learning strategy of an emotional layer and a multimodal layer according to the conversation text embedding vector, the conversation visual embedding vector, and the conversation audio embedding vector corresponding to each piece of the multimodal conversation data; construct a cross-entropy loss based on the emotion label and emotion recognition prediction result of each piece of the multimodal conversation data; and train the model until convergence by minimizing a joint loss function of the contrastive learning loss and the cross-entropy loss. The emotion recognition module is used to use the multi-round question and answer information to be recognized as the input of the converged model and obtain the emotion recognition results of each multimodal dialogue data in the multi-round question and answer information.

8. The knowledge-enhanced product question-and-answer community user conversation emotion recognition system according to claim 7, characterized in that: It also includes a dictionary construction module for constructing the user personalized sentiment dictionary, and the construction process includes: The manually annotated sentiment dictionary NRC-VAD Chinese version is selected as the basic sentiment dictionary, which is expressed as Where (V i , A i , D i ) is the word w i The features of represent the corresponding VAD emotional state vector, which respectively represent the scores of activation, pleasure and dominance in three dimensions as V i , A i , D i , and the value range of the three is between 0 and 1; Crawl the historical Q&A information of all users in the current product Q&A community, build a Q&A corpus for the target domain, remove stop words to obtain a domain dictionary W, and use this dictionary to train a Word2Vec model to map each word in W to a high-dimensional vector containing the word's semantic information. Use cosine similarity to measure the semantic similarity between any two words. For words in W, either belonging to the dictionary L0 or not belonging to the basic sentiment dictionary L0, through the semantic similarity between any two words and several similar words, if there is a similar word belonging to the dictionary L0, then the word is regarded as a potential sentiment word and added to the potential sentiment word set P; For each emotion word in the set P, define the algorithm f to map its feature representation into a VAD emotion state vector; if the three dimensions of the VAD emotion state vector are all greater than 0, then add the emotion word and the corresponding VAD emotion state vector to the dictionary L0, and record the expanded domain emotion dictionary as L new ; Collect the current user's historical question and answer information and treat it as a document. Remove stop words from the document and then perform word segmentation. Use TF-IDF method to extract all keywords from preprocessed documents; If the keyword is in the expanded domain sentiment dictionary L new Then find the VAD emotional state vector corresponding to the keyword; if the keyword is not in the expanded domain emotional dictionary L new In the case of , a zero vector is used instead; The TF-IDF value of the keyword is used as a weight to multiply the VAD emotional state vector or the zero vector to obtain the corresponding preference knowledge embedding vector; Each keyword and the corresponding preference knowledge embedding vector are summarized to construct the user personalized sentiment dictionary.

9. A storage medium, characterized in that: It stores a computer program for identifying emotions in user conversations in a product question-and-answer community for knowledge enhancement, wherein the computer program enables a computer to execute the method for identifying emotions in user conversations in a product question-and-answer community as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the product question and answer community user conversation emotion recognition method as described in any one of claims 1 to 6.

Images