Intelligent real-time emotion evaluation method for social media and online text data based on multi-modal knowledge graph

Abstract

The invention discloses an intelligent real-time emotion evaluation method for social media and online text data based on a multi-modal knowledge graph, and the method comprises the steps: firstly obtaining and preprocessing multi-modal original data; secondly, designing single-mode feature extraction of dialogue texts, voices and facial expressions based on BERT-BiLSTM, AST and ResNet-50 pre-training models; furthermore, sharing among multiple modes and private emotion semantic information are considered, and complementation of cross-mode emotion information is promoted by enhancing repeated emotion information and utilizing uniqueness of private information. Meanwhile, a multi-mode knowledge graph method is provided from bottom to top to carry out emotion real-time evaluation. Analyzing user behaviors by using DeepFM and generating user features, and capturing external knowledge layer features in combination with K-BERT; finally, on the basis of a TransformerEncoder structure, cross-modal emotion information interaction and feature fusion are achieved, real-time recognition and classification of multi-modal emotion perception are completed through a Softmax layer, and therefore the purpose of intelligent real-time emotion evaluation is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal knowledge graphs, and in particular to an intelligent real-time sentiment evaluation method for social media and online text data based on multimodal knowledge graphs. Background Art

[0002] Emotion recognition refers to the use of computer technology to identify and judge a person's emotional state. Emotion recognition is of great significance for improving the human-computer interaction experience, especially in human resource management, e-commerce customer service, and promoting personalized recommendations. In practical applications, online emotion assessment based on social media faces many challenges such as diversification, complexity, and dynamism.

[0003] At present, the research on emotion evaluation in social media is mainly based on conversation text. Conversation emotion recognition achieves text classification by assigning an emotion label to each conversation utterance. However, conversation text is usually informal, with frequent topic switching and dynamic changes in context information. Therefore, conversation emotion recognition needs to combine a variety of related information to further supplement and improve semantic information. Voice information, as another important source of information in social media, can be used to supplement conversation information and realize cross-modal information interaction and supplementation between multiple modalities, so as to further improve information relevance and enhance the accuracy and robustness of evaluation.

[0004] The current mainstream emotion assessment methods mostly use mathematical models or machine learning algorithms. These methods usually extract features manually and are easy to implement and explain. However, due to the quality and coverage of the data set, it is difficult to fully utilize contextual semantic information, and the generalization of the test data distribution offset problem is poor. Knowledge graph is a graphical knowledge base used to represent and store structured knowledge, including operations such as knowledge extraction, knowledge fusion, and knowledge reasoning. By converting different types of entities and relationships into low-dimensional numerical vectors, the computational complexity is effectively reduced, which makes it easier to sort out the complex semantic relationships between entities and alleviate the problem of data sparsity, thereby enhancing the generalization ability of the model and taking into account the interpretability of the model. Summary of the invention

[0005] In order to address the shortcomings of existing emotion recognition methods, the purpose of the present invention is to provide an intelligent real-time emotion assessment method for social media and online text data based on multimodal knowledge graphs, which achieves the purpose of intelligent real-time emotion assessment by modeling conversation text, voice and facial expressions through multimodal knowledge graphs.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An intelligent real-time sentiment evaluation method for social media and online text data based on a multimodal knowledge graph, characterized by comprising the following steps:

[0008] S1: Acquire multimodal data of social media and perform data cleaning and preprocessing, wherein the multimodal data includes conversation text, voice and facial expressions;

[0009] S2: Based on the characteristics of unimodal data, unimodal feature extraction for conversation text, speech and facial expressions is performed based on BERT-BiLSTM, AST and ResNet-50 pre-trained models respectively;

[0010] S3: Construct a text-centric multimodal fusion model, consider the shared emotional semantic information between multiple modalities, enhance the expression of key emotional information within the modality through its repeated emotional information, make full use of the unique characteristics of private emotional semantic information derived from each modal data, and effectively realize cross-modal emotional information complementation through private emotional semantic information from different modalities;

[0011] S4: Use a bottom-up approach to build a multimodal knowledge graph. Based on social cognitive theory, use the DeepFM algorithm to analyze user behavior and generate user features, and rely on the K-BERT model to capture external knowledge layer features.

[0012] S5: Based on the TransformerEncoder structure, cross-modal emotional information interaction and feature fusion are realized, and the Softmax layer is input to complete the recognition of multimodal emotion perception, realizing the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, so as to perform intelligent real-time emotion evaluation.

[0013] In S1, multimodal data from social media is obtained and data cleaning and preprocessing are performed, including the following steps:

[0014] By designing the media client automation tool authorization, the background collects chat records in real time and performs data processing and analysis; the adaptive DBSCAN and K-means combination algorithm is used to clean the abnormal data in the collected data; at the same time, considering the huge amount of background data, the clustering time will be extended as the number of data sets increases; by randomly slicing the original data set, splitting it into multiple sub-data sets for calculation, and finally performing weighted average on the clustering results to obtain the final result, thereby reducing the calculation time and improving the efficiency; then, the abnormal data after the combination cleaning is eliminated and the eliminated values ​​are filled by the random forest algorithm. The method guarantees data quality while efficiently storing data, optimizes data cleaning work, and thus realizes efficient multi-source heterogeneous data cleaning work.

[0015] In S2, the unimodal feature extraction of conversation text, speech, and facial expressions based on the BERT-BiLSTM, AST, and ResNet-50 pre-trained models includes the following steps:

[0016] For dialogue text data, BERT-BiLSTM encoding is used to extract contextual information from the discourse. For voice data, AST encoding is used to extract features. For facial expressions, the ResNet-50 pre-trained model is used for feature extraction. The formula is as follows:

[0017] X T =BERT-BiLSTM(Text)

[0018] X A =AST(Audio)

[0019] X P =ResNet-50(Picture)

[0020] Among them, X T ∈R l1 ×768,X T represents the extracted text feature vector sequence, X A ∈R l2 ×768,X A represents the extracted speech feature vector sequence, X P ∈R l3 ×2048,X P Represents the extracted facial expression feature vector sequence; R is a real number sequence, l1; l2; l3 are the lengths of text, speech and facial expression image sequences respectively.

[0021] In S3, a multimodal fusion model is constructed, taking into account the shared emotional semantic information between multiple modalities, enhancing the expression of key emotional information within the modality through its repeated emotional information, making full use of the unique characteristics of private emotional semantic information derived from each modal data, and effectively realizing cross-modal emotional information complementarity through private emotional semantic information from different modalities. The specific method of constructing a multimodal fusion model is as follows:

[0022] The shared emotion semantics calculation module uses the shared emotion semantics of the dialogue text signal and the speech signal to enhance the expression of the key emotion information of the text;

[0023] Calculate the text information sequence T = [t1, t2, ..., t lT ]’s attention weight wight={w1,w2,…,w lT} and arrange them from large to small. The semantic sentiment reflected in the text information sequence is quantified and expressed in the form of key-value pairs. The formula is as follows:

[0024]

[0025] Among them, w i Indicates the tth text information sequence i The normalized attention weight of the item; Key represents the semantic sentiment key-value pair;

[0026] Calculate the speech feature vector sequence X for each word pair in the text sequence A and facial expression feature vector sequence X P The similarity score S T→I ; I∈{A,P}, use the Softmax function to calculate the attention weight, and use the cross-modal shared mask matrix to obtain the shared emotional semantic feature vector C I , and splice the features according to the text dimension, use the self-attention mechanism layer to mine the important emotional information features and strengthen the key features to obtain F m ; Then it is sent to LSTM to add time sequence information and obtain the final text sentiment feature F t , the formula is as follows: m =Self_Attention([C A ; C P ;X T ])

[0027] F t =LSTM(F m )

[0028] Among them, F m Represents the emotional information features after splicing; C A and C P Represent the shared emotional semantic feature vectors from speech and facial expressions respectively; F t Represents the text sentiment feature with temporal information, F t ∈R 1×d ; d is the output dimension of the hidden layer of the LSTM network;

[0029] Private emotional semantic computing focuses on the emotional information within the speech modality and facial expression modality. It adopts the Encoder-Decoder framework structure. After the feature sequence is obtained through pre-trained model encoding, it is sent to the Decoder module to decode and reorganize the feature sequence to obtain private semantic emotional features.

[0030] The self-attention mechanism layer is used to mine the important emotional features in facial expressions and voices to increase the weight of key emotional features. The input of the self-attention mechanism is obtained based on the linear matrix. The facial expressions are integrated through LSTM to obtain the high-level emotional feature sequence X. I ' AttentionAnd add time sequence information to obtain the private emotional semantic features F within the facial expression and voice modality I , the formula is as follows:

[0031]

[0032] F I =LSTM(X I ' Attention )

[0033] Among them, X I ' Attention Represents the sequence of high-level emotional features of facial expressions and voice; F I Represents private emotional semantic features within facial expressions and speech modalities; Query I ' is the characteristic element in the sequence of facial expression and voice high-level emotional features; Key I ' and Value I ' is the key-value pair of semantic sentiment information corresponding to the feature element; The purpose is to prevent the partial derivative from being 0 when the gradient propagates in the reverse direction.

[0034] In S4, in order to solve the problems of data heterogeneity and difficulty in complementary fusion in the process of real-time emotion evaluation, a method for constructing a multimodal knowledge graph is proposed. The bottom-up construction method is adopted. Based on social cognitive theory, the DeepFM algorithm is used to analyze user behavior to generate user features, and the K-BERT model is used to capture the external knowledge layer features. The details are as follows:

[0035] The DeepFM algorithm is used to analyze the conversion of continuous features and the learning efficiency problem caused by data sparsity. The continuous variables are discretized and converted using one-hot encoding. The DeepFM algorithm is used to extract low-order feature combinations, and their relevance is measured by the inner product of the features. The DNN model learns high-order feature combinations and shares the embedding layer with DeepFM to enhance feature interaction and further capture user interaction attribute characteristics.

[0036] Through the K-BERT model, knowledge enhancement is performed and the knowledge graph is integrated into the language representation to improve the interpretability of the K-BERT model. At the knowledge layer, the K-BERT model uses knowledge queries to match entities in the graph and injects triple information into the text to form a sentence tree rich in background knowledge. The embedding layer uses the pre-trained language model BERT-BiLSTM to map the semantic information of the sentence tree to the vector space to obtain token embedding. The sentence tree uses soft-position embedding and is written in a way that the trunk is continuously numbered and the branches are gradually numbered. Segment embedding is used to mark the division of each sentence. The visual layer introduces a visible matrix to control the connection between words and determine whether they are on the same branch; finally, the embedded representation and the visible matrix are passed to the Mask-Transformer encoding layer composed of multiple Mask-self-attention blocks stacked together to generate an external knowledge layer feature vector representation containing rich semantic information to achieve a deep understanding of the text.

[0037] In S5, the TransformerEncoder structure is used to realize cross-modal emotional information interaction and feature fusion, and the Softmax layer is input to complete the recognition of multi-modal emotional perception, so as to realize the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, so as to perform intelligent real-time emotional evaluation, as follows:

[0038] In the fusion mechanism, the obtained enhanced text feature F T , speech and facial private emotional semantic features F A and F P , forming a new feature sequence M = {F T ,F A ,F P}, the modal fusion module is based on the multi-head self-attention mechanism and optimizes the TransformerEncoder structure; after cross-modal information interaction in multiple different subspaces, the modal feature sequence F is obtained temp1 , the modal feature sequence F temp1 Send it to the feedforward neural network for nonlinear mapping to obtain the modal feature sequence F temp2 , F' is the new feature sequence, F' = {F T ',F A ',F P '}, the formula is as follows:

[0039] F temp1 =MultiHead Self_Attention(M)

[0040] F temp2 =max(0,F temp1 W1+b1)W2+b2

[0041] F′=Layer Norm(F temp1 +F temp2 )

[0042] In the formula, F temp1 represents the initial multimodal feature sequence; M is a sequence containing text, speech and facial private emotional semantic features; F temp2 represents the multimodal feature sequence after fitting the data feature distribution; W1, W2 are linear transformation matrices; b1, b2 are biases; F' represents the new feature sequence after the three modalities are fully integrated, F T ',F A ' and F P 'F i 'Text, speech and facial emotion semantic features in;

[0043] Send F' to the fully connected layer, and then contact the feature vectors of the knowledge layer, user layer and external knowledge layer to get the final vector Z. Input Z into the multi-head attention mechanism for learning, and use the Softmax layer to implement text classification to output the sentiment prediction result Z. The formula is as follows: Z = Linear[Concat(F T ′,F A ′,F P ′)].

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] 1. The present invention addresses the problem of outliers in the collected raw data and designs a data cleaning model based on the adaptive DBSCAN and K-means combined algorithm. The original data set is randomly sliced ​​to obtain multiple sub-data sets, and the clustering results are weighted averaged to obtain the final result. This design link can reduce the recognition time and significantly improve the efficiency and accuracy of subsequent single-modal recognition.

[0046] 2. The present invention is based on a mature and efficient unimodal feature extraction method. BERT-BiLSTM, AST and ResNet-50 pre-trained models are used to extract unimodal features of dialogue text, speech and facial expressions, ensuring the reliability and accuracy of the extracted modal features.

[0047] 3. The present invention constructs an intelligent emotion assessment method for shared-private information. By enhancing repeated emotion information and utilizing the uniqueness of private information, the complementarity of cross-modal emotion information is promoted, thereby improving the accuracy of emotion assessment. In addition, a bottom-up multimodal knowledge graph is used to conduct real-time emotion assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a diagram showing the emotion assessment framework based on the multimodal knowledge graph of the present invention. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present application, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0050] like Figure 1 As shown, the present invention provides an intelligent real-time emotion evaluation method for social media and online text data based on a multimodal knowledge graph, characterized in that it includes the following steps:

[0051] S1: Acquire multimodal data of social media and perform data cleaning and preprocessing, wherein the multimodal data includes conversation text, voice and facial expressions;

[0052] S2: Based on the characteristics of unimodal data, unimodal feature extraction for conversation text, speech and facial expressions is performed based on BERT-BiLSTM, AST and ResNet-50 pre-trained models respectively;

[0053] S3: Construct a text-centric multimodal fusion model, consider the shared emotional semantic information between multiple modalities, enhance the expression of key emotional information within the modality through its repeated emotional information, make full use of the unique characteristics of private emotional semantic information derived from each modal data, and effectively realize cross-modal emotional information complementation through private emotional semantic information from different modalities;

[0054] S4: Use a bottom-up approach to build a multimodal knowledge graph. Based on social cognitive theory, use the DeepFM algorithm to analyze user behavior and generate user features, and rely on the K-BERT model to capture external knowledge layer features.

[0055] S5: Based on the TransformerEncoder structure, cross-modal emotional information interaction and feature fusion are realized, and the Softmax layer is input to complete the recognition of multimodal emotion perception, realizing the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, so as to perform intelligent real-time emotion evaluation.

[0056] In S1, the multimodal data of social media is obtained and data cleaning and preprocessing are performed, including the following steps: Multimodal experimental data includes conversation text, voice and facial expressions. Through the design of media client automation tool authorization, the background collects chat records in real time and performs data processing and analysis. Since the data collected in the background is susceptible to noise and inconsistent data, low-quality data will lead to low-quality mining results, affecting subsequent evaluation work.

[0057] In the embodiment, the specific operation is as follows: Adopt adaptive DBSCAN and K-means combined algorithm to clean abnormal data in collected data. Considering the huge amount of background data, the clustering time will increase with the increase of the number of data sets. By randomly slicing the original data set, splitting it into multiple sub-data sets for calculation, and finally performing weighted averaging on the clustering results to obtain the final result, the calculation time is reduced and the efficiency is improved. Then, the abnormal data after combined cleaning is eliminated and the eliminated values ​​are filled by random forest algorithm. The method optimizes data cleaning work while efficiently storing data and ensuring data quality, thereby realizing efficient multi-source heterogeneous data cleaning work.

[0058] In S2, unimodal feature extraction of conversation text, speech and facial expressions is performed based on BERT-BiLSTM, AST and ResNet-50 pre-trained models, including the following steps: unimodal feature extraction of conversation text based on BERT-BiLSTM, unimodal feature extraction of speech based on AST model, and facial expression feature extraction based on ResNet-50.

[0059] In the embodiment, the specific method of feature extraction is as follows: for dialogue text data, BERT-BiLSTM encoding is used to extract context information from the speech, for voice data, AST encoding is used to extract features, and for facial expressions, ResNet-50 pre-trained model is used for feature extraction, and the formula is as follows: X T =BERT-BiLSTM(Text)

[0060] X A =AST(Audio)

[0061] X P =ResNet-50(Picture)

[0062] Among them, X T ∈R l1 ×768,X T represents the extracted text feature vector sequence, X A ∈R l2 ×768,X A represents the extracted speech feature vector sequence, X P ∈R l3 ×2048,X P represents the extracted facial expression feature vector sequence; R is a real number sequence. l1; l2; l3 are the lengths of text, speech and facial expression image sequences respectively.

[0063] In S3, a multimodal fusion model is constructed, taking into account the shared emotional semantic information among multiple modalities, enhancing the expression of key emotional information within the modality through its repeated emotional information, making full use of the unique characteristics of private emotional semantic information derived from each modal data, and effectively realizing cross-modal emotional information complementarity through private emotional semantic information from different modalities.

[0064] In the embodiment, the specific method of constructing the multimodal fusion model is as follows: the shared emotion semantics calculation module uses the shared emotion semantics information of the dialogue text signal and the speech signal to enhance the expression of the key emotion information of the text, and calculates the text information sequence T = [t1, t2, ..., t lT ]’s attention weight wight={w1,w2,…,w lT} and arrange them from large to small. The semantic sentiment reflected in the text information sequence is quantified and expressed in the form of key-value pairs. The formula is as follows:

[0065]

[0066] Among them, w i Indicates the tth text information sequence i The normalized attention weight of the item; Key represents the semantic sentiment key-value pair;

[0067] Calculate the value of X for each word pair in the text sequence A and X P The similarity score S T→I ; I∈{A,P}; Use Softmax function to calculate the attention weight W T→I , using the cross-modal shared mask matrix to obtain the shared emotional semantic feature vector C I , and splice the features according to the text dimension, use the self-attention mechanism layer to mine the important emotional information features and strengthen the key features to obtain F m , and then sent to LSTM to add time sequence information and obtain the final text sentiment feature F t , the formula is as follows:

[0068] F m =Self_Attention([C A ; C P ;X T ])

[0069] F t =LSTM(F m )

[0070] Among them, F m Represents the emotional information features after splicing; C A and C PRepresent the shared emotional semantic feature vectors from speech and facial expressions respectively; F t Represents the text sentiment feature with temporal information, F t ∈R 1×d ; d is the output dimension of the hidden layer of the LSTM network.

[0071] Private emotional semantic computing focuses on the emotional information within the speech modality and facial expression modality. It adopts the Encoder-Decoder framework structure. After the feature sequence is obtained through pre-training model encoding, it is sent to the Decoder module to decode and reorganize the feature sequence to obtain private semantic emotional features. The self-attention mechanism layer is used to mine the important emotional features within the expression and speech to increase the weight of the key emotional features. The input of the self-attention mechanism is obtained based on the linear matrix. The facial expressions are integrated through LSTM to obtain the high-level emotional feature sequence X I ' Attention And add time sequence information to obtain the private emotional semantic features F within the facial expression and voice modality I , the formula is as follows:

[0072]

[0073] F I =LSTM(X I ' Attention )

[0074] Among them, X I ' Attention Represents the sequence of high-level emotional features of facial expressions and voice; F I Represents private emotional semantic features within facial expressions and speech modalities; Query I ' is the characteristic element in the sequence of facial expression and voice high-level emotional features; Key I ' and Value I ' is the key-value pair of semantic sentiment information corresponding to the feature element; The purpose is to prevent the partial derivative from being 0 when the gradient propagates in the reverse direction.

[0075] In S4, in order to solve the problems of data heterogeneity and difficulty in complementary fusion in the process of real-time emotion evaluation, a method for constructing a multimodal knowledge graph is proposed. The bottom-up construction method is adopted. Based on social cognitive theory, the DeepFM algorithm is used to analyze user behavior to generate user features, and the K-BERT model is used to capture the external knowledge layer features. The details are as follows:

[0076] The DeepFM algorithm is used to analyze the conversion of continuous features and the learning efficiency problems caused by data sparsity. The continuous variables are discretized and converted using one-hot encoding. The DeepFM algorithm is used to extract low-order feature combinations, and their correlation is measured by the inner product of the features. The DNN model learns high-order feature combinations and shares the embedding layer with DeepFM to enhance feature interaction and further capture user interaction attribute characteristics.

[0077] Through the K-BERT model, knowledge enhancement is performed and the knowledge graph is integrated into the language representation to improve the interpretability of the K-BERT model. At the knowledge layer, the K-BERT model uses knowledge queries to match entities in the graph and injects triple information into the text to form a sentence tree rich in background knowledge. The embedding layer uses the pre-trained language model BERT-BiLSTM to map the semantic information of the sentence tree to the vector space to obtain token embedding. The sentence tree uses soft-position embedding and is written in a way that the trunk is continuously numbered and the branches are gradually numbered. Segment embedding is used to mark the division of each sentence. The visual layer introduces a visible matrix to control the connection between words and determine whether they are on the same branch; finally, the embedded representation and the visible matrix are passed to the Mask-Transformer encoding layer composed of multiple Mask-self-attention blocks stacked together to generate an external knowledge layer feature vector representation containing rich semantic information to achieve a deep understanding of the text.

[0078] In S5, feature interaction and fusion are performed to carry out intelligent real-time emotion evaluation, including the following steps: based on the Transformer Encoder structure, cross-modal emotion information interaction and feature fusion are realized, and the Softmax layer is input to complete the recognition of multi-modal emotion perception, so as to realize the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, so as to carry out intelligent real-time emotion evaluation;

[0079] In the embodiment, the specific operation is as follows: In the fusion mechanism, the obtained enhanced text feature F T , speech, human face private emotional semantic features F A and F P , forming a new feature sequence M = {F T ,F A ,F P}, the modal fusion module is based on the multi-head self-attention mechanism and optimizes the TransformerEncoder structure. After cross-modal information interaction in multiple different subspaces, the modal feature sequence F is obtained. temp1 , the modal feature sequence F temp1 Send it to the feedforward neural network for nonlinear mapping to obtain the modal feature sequence Ftemp2 , F' is the new feature sequence, F'={F' T ,F' A ,F' P}, the formula is as follows:

[0080] F temp1 =MultiHead Self_Attention(M)

[0081] F temp2 =max(0,F temp1 W1+b1)W2+b2

[0082] F′=Layer Norm(F temp1 +F temp2 )

[0083] In the formula, F temp1 represents the initial multimodal feature sequence; M is a sequence containing text, speech and facial private emotional semantic features; F temp2 represents the multimodal feature sequence after fitting the data feature distribution; W1, W2 are linear transformation matrices; b1, b2 are biases; F' represents the new feature sequence after the three modalities are fully integrated, F T ',F A ' and F P 'F i 'Text, speech and facial emotion semantic features in;

[0084] Send F' to the fully connected layer, and then contact the feature vectors of the knowledge layer, user layer and external knowledge layer to get the final vector Z. Input Z into the multi-head attention mechanism for learning, and use the Softmax layer to implement text classification to output the sentiment prediction result Z. The formula is as follows: Z = Linear[Concat(F t ′,F a ′,F p ′)]

[0085] The present invention realizes an intelligent real-time emotion evaluation method based on social media and online text data of multimodal knowledge graph, which is conducive to improving the accuracy and real-time performance of emotion evaluation, and can effectively integrate multiple data sources (including dialogue text, voice, facial expression and external knowledge layer), thereby enhancing the comprehensive understanding of the user's emotional state. The method of the present invention reduces the complexity of multivariate heterogeneity and cross-modal fusion of data, realizes cross-modal interaction and feature fusion of emotional information through the construction of multimodal knowledge graph and the application of multi-head attention mechanism, enhances the generalization ability and interpretability of the model, and thus improves the recognition accuracy of emotional perception, which has important application value in human resource management, e-commerce customer service, and promoting personalized recommendations.

[0086] The above description of the disclosure of the present invention is provided to enable any person of ordinary skill in the art to implement or use the disclosure of the present invention. It will be apparent to those of ordinary skill in the art that various modifications to the disclosure of the present invention may be made, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the disclosure of the present invention. Therefore, the disclosure of the present invention is not limited to the examples and designs described herein, but is consistent with the broadest range of principles and novel features disclosed herein.

Claims

1. An intelligent real-time sentiment evaluation method for social media and online text data based on multimodal knowledge graph, characterized in that: The following steps are involved: S1: Acquire multimodal data of social media and perform data cleaning and preprocessing, wherein the multimodal data includes conversation text, voice and facial expressions; S2: Based on the characteristics of unimodal data, unimodal feature extraction for conversation text, speech and facial expressions is performed based on BERT-BiLSTM, AST and ResNet-50 pre-trained models respectively; S3: Construct a text-centric multimodal fusion model, consider the shared emotional semantic information between multiple modalities, enhance the expression of key emotional information within the modality through its repeated emotional information, make full use of the unique characteristics of private emotional semantic information derived from each modal data, and effectively realize cross-modal emotional information complementation through private emotional semantic information from different modalities; S4: Use a bottom-up approach to build a multimodal knowledge graph. Based on social cognitive theory, use the DeepFM algorithm to analyze user behavior and generate user features, and rely on the K-BERT model to capture external knowledge layer features. S5: Based on the TransformerEncoder structure, cross-modal emotional information interaction and feature fusion are realized, and the Softmax layer is input to complete the recognition of multimodal emotion perception, realizing the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, so as to perform intelligent real-time emotion evaluation.

2. According to claim 1, a method for intelligent real-time emotion evaluation of social media and online text data based on multimodal knowledge graph, characterized in that: In S1, multimodal data from social media is obtained and data cleaning and preprocessing are performed, including the following steps: By designing the media client automation tool authorization, the background collects chat records in real time and performs data processing and analysis; the adaptive DBSCAN and K-means combination algorithm is used to clean the abnormal data in the collected data; at the same time, considering the huge amount of background data, the clustering time will be extended as the number of data sets increases; by randomly slicing the original data set, splitting it into multiple sub-data sets for calculation, and finally performing weighted average on the clustering results to obtain the final result, thereby reducing the calculation time and improving the efficiency; then, the abnormal data after the combination cleaning is eliminated and the eliminated values ​​are filled by the random forest algorithm. The method guarantees data quality while efficiently storing data, optimizes data cleaning work, and thus realizes efficient multi-source heterogeneous data cleaning work.

3. According to claim 1, the intelligent real-time emotion evaluation method for social media and online text data based on multimodal knowledge graph is characterized in that: In S2, the unimodal feature extraction of conversation text, speech, and facial expressions based on the BERT-BiLSTM, AST, and ResNet-50 pre-trained models includes the following steps: For dialogue text data, BERT-BiLSTM encoding is used to extract contextual information from the discourse. For voice data, AST encoding is used to extract features. For facial expressions, the ResNet-50 pre-trained model is used for feature extraction. The formula is as follows: X T =BERT-BiLSTM(Text) X A =AST(Audio) X P =ResNet-50(Picture) Among them, X T ∈R l1 ×768,X T represents the extracted text feature vector sequence, X A ∈R l2 ×768,X A represents the extracted speech feature vector sequence, X P ∈R l3 ×2048,X P Represents the extracted facial expression feature vector sequence; R is a real number sequence, l1; l2; l3 are the lengths of text, speech and facial expression image sequences respectively.

4. According to claim 1, the intelligent real-time emotion evaluation method for social media and online text data based on multimodal knowledge graph is characterized in that: In S3, a multimodal fusion model is constructed, taking into account the shared emotional semantic information between multiple modalities, enhancing the expression of key emotional information within the modality through its repeated emotional information, making full use of the unique characteristics of private emotional semantic information derived from each modal data, and effectively realizing cross-modal emotional information complementarity through private emotional semantic information from different modalities. The specific method of constructing a multimodal fusion model is as follows: The shared emotion semantics calculation module uses the shared emotion semantics of the dialogue text signal and the speech signal to enhance the expression of the key emotion information of the text; Calculate the text information sequence T = [t1, t2, ..., t lT ]’s attention weight wight={w1,w2,…,w lT } and arrange them from large to small. The semantic sentiment reflected in the text information sequence is quantified and expressed in the form of key-value pairs. The formula is as follows: Among them, w i Indicates the tth text information sequence i The normalized attention weight of the item; Key represents the semantic sentiment key-value pair; Calculate the speech feature vector sequence X for each word pair in the text sequence A and facial expression feature vector sequence X P The similarity score S T→I ; I∈{A,P}, use the Softmax function to calculate the attention weight, and use the cross-modal shared mask matrix to obtain the shared emotional semantic feature vector C I , and splice the features according to the text dimension, use the self-attention mechanism layer to mine the important emotional information features and strengthen the key features to obtain F m ; Then it is sent to LSTM to add time sequence information and obtain the final text sentiment feature F t , the formula is as follows: F m =Self_Attention([C A ;C P ;X T ]) F t =LSTM(F m ) Among them, F m Represents the emotional information features after splicing; C A and C P Represent the shared emotional semantic feature vectors from speech and facial expressions respectively; F t Represents the text sentiment feature with temporal information, F t ∈R 1×d ; d is the output dimension of the hidden layer of the LSTM network; Private emotional semantic computing focuses on the emotional information within the speech modality and facial expression modality. It adopts the Encoder-Decoder framework structure. After the feature sequence is obtained through pre-trained model encoding, it is sent to the Decoder module to decode and reorganize the feature sequence to obtain private semantic emotional features. The self-attention mechanism layer is used to mine the important emotional features in facial expressions and voices to increase the weight of key emotional features. The input of the self-attention mechanism is obtained based on the linear matrix. The facial expressions are integrated through LSTM to obtain the high-level emotional feature sequence X. I ' Attention And add time sequence information to obtain the private emotional semantic features F within the facial expression and voice modality I , the formula is as follows: F I =LSTM(X I ' Attention ) Among them, X I ' Attention Represents the sequence of high-level emotional features of facial expressions and voice; F I Represents private emotional semantic features within facial expressions and speech modalities; Query I ' is the characteristic element in the sequence of facial expression and voice high-level emotional features; Key I ' and Value I ' is the key-value pair of semantic sentiment information corresponding to the feature element; The purpose is to prevent the partial derivative from being 0 when the gradient propagates in the reverse direction.

5. The intelligent real-time emotion evaluation method for social media and online text data based on multimodal knowledge graph according to claim 1 is characterized in that: In S4, in order to solve the problems of data heterogeneity and difficulty in complementary fusion in the process of real-time emotion evaluation, a method for constructing a multimodal knowledge graph is proposed. The bottom-up construction method is adopted. Based on social cognitive theory, the DeepFM algorithm is used to analyze user behavior to generate user features, and the K-BERT model is used to capture the external knowledge layer features. The details are as follows: The DeepFM algorithm is used to analyze the conversion of continuous features and the learning efficiency problem caused by data sparsity. The continuous variables are discretized and converted using one-hot encoding. The DeepFM algorithm is used to extract low-order feature combinations, and their relevance is measured by the inner product of the features. The DNN model learns high-order feature combinations and shares the embedding layer with DeepFM to enhance feature interaction and further capture user interaction attribute characteristics. Through the K-BERT model, knowledge enhancement is performed and the knowledge graph is integrated into the language representation to improve the interpretability of the K-BERT model. At the knowledge layer, the K-BERT model uses knowledge queries to match entities in the graph and injects triple information into the text to form a sentence tree rich in background knowledge. The embedding layer uses the pre-trained language model BERT-BiLSTM to map the semantic information of the sentence tree to the vector space to obtain token embedding. The sentence tree uses soft-position embedding and is written in a way that the trunk is continuously numbered and the branches are gradually numbered. Segment embedding is used to mark the division of each sentence. The visual layer introduces a visible matrix to control the connection between words and determine whether they are on the same branch; finally, the embedded representation and the visible matrix are passed to the Mask-Transformer encoding layer composed of multiple Mask-self-attention blocks stacked together to generate an external knowledge layer feature vector representation containing rich semantic information to achieve a deep understanding of the text.

6. The intelligent real-time emotion evaluation method for social media and online text data based on multimodal knowledge graph according to claim 1 is characterized in that: In S5, the Transformer Encoder structure is used to realize cross-modal emotional information interaction and feature fusion, and the Softmax layer is input to complete the recognition of multi-modal emotional perception, so as to realize the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, so as to perform intelligent real-time emotional evaluation, as follows: In the fusion mechanism, the obtained enhanced text feature F T , speech and facial private emotional semantic features F A and F P , forming a new feature sequence M = {F T ,F A ,F P }, the modal fusion module is based on the multi-head self-attention mechanism and optimizes the TransformerEncoder structure; after cross-modal information interaction in multiple different subspaces, the modal feature sequence F is obtained temp1 , the modal feature sequence F temp1 Send it to the feedforward neural network for nonlinear mapping to obtain the modal feature sequence F temp2 , F' is the new feature sequence, F' = {F T ',F A ',F P '}, the formula is as follows: F temp1 =MultiHead Self_Attention(M) F temp2 =max(0,F temp1 W1+b1)W2+b2 F′=Layer Norm(F temp1 +F temp2 ) In the formula, F temp1 represents the initial multimodal feature sequence; M is a sequence containing text, speech and facial private emotional semantic features; F temp2 represents the multimodal feature sequence after fitting the data feature distribution; W1, W2 are linear transformation matrices; b1, b2 are biases; F' represents the new feature sequence after the three modalities are fully integrated, F T ',F A ' and F P 'F i 'Text, speech and facial emotion semantic features in; Send F' to the fully connected layer, and then contact the feature vectors of the knowledge layer, user layer and external knowledge layer to get the final vector Z. Input Z into the multi-head attention mechanism for learning, and use the Softmax layer to implement text classification to output the sentiment prediction result Z. The formula is as follows: Z=Linear[Concat(F T ′,F A ′,F P ′)]。

Images