Intelligent real-time sentiment evaluation method of social media and online text data based on multi-modal knowledge graph

Abstract

The application discloses a kind of based on multimodal knowledge graph's social media and online text data intelligent real-time emotion evaluation method, first, obtains and pre-processes multimodal original data;Second, design based on BERT-BiLSTM, AST and ResNet-50 pre-training model's single mode feature extraction of dialogue text, voice and facial expression;Further, consider the shared and private emotional semantic information between multimodal, by enhancing repeated emotional information and using the uniqueness of private information, promote the complement of cross-modal emotional information.Simultaneously, from bottom to top, the method of multimodal knowledge graph is proposed to carry out emotional real-time evaluation.DeepFM is used to analyze user behavior and generate user features, and K-BERT is used to capture external knowledge layer features.Finally, based on the structure of TransformerEncoder, cross-modal emotional information interaction and feature fusion are realized, and real-time recognition and classification of multimodal emotion perception are completed through the Softmax layer, thereby achieving the purpose of intelligent real-time emotion evaluation.

Description

Technical Field

[0001] This invention relates to the field of multimodal knowledge graph technology, and in particular to an intelligent real-time sentiment assessment method based on multimodal knowledge graphs for social media and online text data. Background Technology

[0002] Emotion recognition refers to the use of computer technology to identify and judge a person's emotional state. Emotion recognition is of great significance for improving human-computer interaction, particularly in human resource management, e-commerce customer service, and promoting personalized recommendations. In practical applications, conducting online emotion assessments based on social media faces numerous challenges, including diversification, complexity, and dynamism.

[0003] Current research on emotion assessment in social media primarily focuses on dialogue text. Dialogue emotion recognition classifies text by assigning an emotion label to each utterance. However, dialogue texts are typically informal, characterized by frequent topic shifts and dynamically changing contextual information. Therefore, dialogue emotion recognition needs to incorporate various relevant information to further supplement and refine semantic information. Voice information, as another important source of information in social media, can be used to supplement dialogue information, enabling cross-modal information interaction and supplementation, thereby further improving information relevance and enhancing the accuracy and robustness of the assessment.

[0004] Current mainstream sentiment assessment methods mostly employ mathematical models or machine learning algorithms. These methods typically extract features manually, making them easy to implement and interpret. However, limited by the quality and coverage of the dataset, they struggle to fully utilize contextual semantic information and exhibit poor generalization ability to address data distribution bias. Knowledge graphs, on the other hand, are graphical knowledge bases used to represent and store structured knowledge, including operations such as knowledge extraction, knowledge fusion, and knowledge reasoning. By transforming different types of entities and relationships into low-dimensional numerical vectors, they effectively reduce computational complexity, facilitate the understanding of complex semantic relationships between entities, alleviate data sparsity problems, and thus enhance the model's generalization ability while maintaining interpretability. Summary of the Invention

[0005] To address the shortcomings of existing emotion recognition methods, the present invention aims to provide an intelligent real-time emotion assessment method based on multimodal knowledge graphs for social media and online text data. This method achieves intelligent real-time emotion assessment by modeling dialogue text, speech, and facial expressions using multimodal knowledge graphs.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An intelligent real-time sentiment assessment method based on multimodal knowledge graphs for social media and online text data, characterized by the following steps:

[0008] S1: Acquire multimodal data from social media and perform data cleaning and preprocessing, wherein the multimodal data includes dialogue text, voice, and facial expressions;

[0009] S2: Based on the characteristics of single-modal data, single-modal feature extraction for dialogue text, speech and facial expressions is carried out using BERT-BiLSTM, AST and ResNet-50 pre-trained models respectively;

[0010] S3: Construct a text-centric multimodal fusion model that considers shared sentiment semantic information among multiple modalities, enhances the expression of key sentiment information within a modality through repeated sentiment information, fully utilizes the unique characteristics of private sentiment semantic information derived from each modality's data, and effectively achieves cross-modal sentiment information complementarity through private sentiment semantic information from different modalities.

[0011] S4: A bottom-up approach is adopted to construct a multimodal knowledge graph. Based on social cognition theory, the DeepFM algorithm is used to analyze user behavior and generate user features, and the K-BERT model is used to capture features of the external knowledge layer.

[0012] S5: Based on the TransformerEncoder structure, it realizes cross-modal emotional information interaction and feature fusion. The input to the Softmax layer completes the recognition of multimodal emotion perception, realizing the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, thereby conducting intelligent real-time emotion assessment.

[0013] In S1, multimodal data from social media is acquired and cleaned and preprocessed, including the following steps:

[0014] By designing an automated tool for media client authorization, the backend collects chat logs in real time for data processing and analysis. An adaptive DBSCAN and K-means combined algorithm is used to clean outlier data from the collected data. Considering the large volume of backend data, the clustering time increases with the number of data points. The original dataset is randomly sliced ​​into multiple subsets for computation, and the clustering results are then weighted and averaged to obtain the final result, thus reducing computation time and improving efficiency. Then, outlier data is removed after cleaning, and the removed values ​​are filled in using a random forest algorithm. This method efficiently stores data while ensuring data quality and optimizing data cleaning, thereby achieving efficient cleaning of multi-source heterogeneous data.

[0015] In S2, the extraction of single-modal features of dialogue text, speech, and facial expressions based on BERT-BiLSTM, AST, and ResNet-50 pre-trained models includes the following steps:

[0016] For dialogue text data, BERT-BiLSTM encoding is used to extract contextual information from the utterance; for speech data, AST encoding is used to extract features; and for facial expressions, a ResNet-50 pre-trained model is used for feature extraction. The formulas are as follows:

[0017] X T =BERT-BiLSTM(Text)

[0018] X A =AST(Audio)

[0019] X P =ResNet-50(Picture)

[0020] Among them, X T ∈R l1 ×768, X T X represents the extracted text feature vector sequence. A ∈R l2 ×768, X A X represents the extracted speech feature vector sequence. P ∈R l3 ×2048, X P R represents the extracted facial expression feature vector sequence; l1; l2; l3 are the sequences of text, speech, and facial expression images, respectively.

[0021] In S3, a multimodal fusion model is constructed, considering the shared sentiment semantic information among multiple modalities. The expression of key sentiment information within a modality is enhanced through repeated sentiment information. The unique characteristics of private sentiment semantic information originating from each modality's data are fully utilized. Cross-modal sentiment information complementarity is effectively achieved through private sentiment semantic information from different modalities. The specific method for constructing the multimodal fusion model is as follows:

[0022] The shared sentiment semantics computing module uses shared sentiment semantics information from dialogue text signals and speech signals to enhance the expression of key sentiment information in the text;

[0023] Calculate the text information sequence T = [t1, t2, ..., t lT The attention weights are wight = {w1, w2, ..., w lT The text information sequences are then arranged from largest to smallest. The semantic sentiment embodied in the text information sequences is quantified and expressed as key-value pairs, as shown in the following formula:

[0024]

[0025] Among them, w i Represents the t-th word in the text information sequence i Normalized attention weights for each item; Key represents semantic sentiment key-value pairs;

[0026] Calculate the speech feature vector sequence X for each word in the text sequence. A and facial expression feature vector sequence X P similarity score S T→I For I∈{A,P}, the attention weights are calculated using the Softmax function, and the shared sentiment semantic feature vector C is obtained using the cross-modal shared mask matrix. I The features are then concatenated along the text dimension, and a self-attention mechanism layer is used to extract important sentiment information features and enhance key features to obtain F. m Then, the data is fed into an LSTM to add temporal information and obtain the final text sentiment feature F. t The formula is as follows: F m =Self_Attention([C A C P ;X T ])

[0027] F t =LSTM(F m )

[0028] Among them, F m Indicates the emotional information characteristics after splicing; C A and C P F represents the shared emotional semantic feature vectors from speech and facial expressions, respectively; t F represents the sentiment features of text with temporal information. t ∈R 1×d d represents the hidden layer output dimension of the LSTM network.

[0029] Private sentiment semantic computing focuses on the sentiment information within speech modalities and facial expression modalities. It adopts an Encoder-Decoder framework structure. After obtaining the feature sequence through pre-trained model encoding, it is fed into the Decoder module to decode and reconstruct the feature sequence to obtain private sentiment features.

[0030] This paper utilizes a self-attention mechanism layer to mine important emotional features within facial expressions and speech to increase the weight of key emotional features. The input to the self-attention mechanism is obtained based on a linear matrix, and facial expressions are integrated using an LSTM to obtain a high-level emotional feature sequence X. I ' AttentionAnd add temporal information to obtain private emotional semantic features F within facial expressions and speech modalities. I The formula is as follows:

[0031]

[0032] F I =LSTM(X) I ' Attention )

[0033] Among them, X I ' Attention Represents a sequence of high-level emotional features in facial expressions and speech; F I Represents private emotional semantic features within facial expressions and speech modalities; Query I 'Feature elements in high-level emotion feature sequences of facial expressions and speech; Key I 'and Value I ' represents the semantic sentiment information key-value pair corresponding to the feature element; Its purpose is to prevent the partial derivatives from being zero when the gradient propagates in the reverse direction.

[0034] In S4, to address the challenges of diverse and heterogeneous data and the difficulty in integrating complementary data in real-time emotion assessment, a method for constructing a multimodal knowledge graph is proposed. This method employs a bottom-up approach, based on social cognition theory, using the DeepFM algorithm to analyze user behavior and generate user features, and relying on the K-BERT model to capture external knowledge layer features, as detailed below:

[0035] The DeepFM algorithm is used to analyze the transformation of continuous features and the learning efficiency problem caused by data sparsity. The continuous variables are discretized and transformed using one-hot encoding. The DeepFM algorithm is used to extract low-order feature combinations and measure their correlation by the inner product of the features. The DNN model learns high-order feature combinations and shares the embedding layer with DeepFM to enhance feature interaction and further capture user interaction attribute features.

[0036] Knowledge augmentation is achieved by integrating the knowledge graph into the language representation using the K-BERT model to enhance its interpretability. At the knowledge layer, the K-BERT model uses knowledge queries to match entities in the graph and injects triplet information into the text, forming a sentence tree rich in background knowledge. The embedding layer uses the pre-trained language model BERT-BiLSTM to map the semantic information of the sentence tree to a vector space, obtaining token embeddings. The sentence tree uses soft-position embedding, written according to continuous numbering of the main branch and progressive numbering of branches. Segment embeddings are used to label the division of each sentence. The visual layer introduces a visibility matrix to control the connections between words and determine whether they are on the same branch. Finally, the embedding representation and the visibility matrix are fed into a Mask-Transformer encoding layer composed of multiple stacked Mask-self-attention blocks to generate an external knowledge layer feature vector representation containing rich semantic information, achieving a deeper understanding of the text.

[0037] In S5, the TransformerEncoder-based structure enables cross-modal emotional information interaction and feature fusion. The input to the Softmax layer completes the recognition of multimodal emotion perception, realizing the integration and interconnection of dialogue text, voice information, facial expressions, and external knowledge layers, thereby performing intelligent real-time emotion assessment, as detailed below:

[0038] In the fusion mechanism, the obtained enhanced text features F T Voice and facial private emotional semantic features F A and F P This constitutes a new feature sequence M = {F} T ,F A ,F P The modality fusion module is based on a multi-head self-attention mechanism and optimizes the TransformerEncoder structure; after cross-modal information interaction in multiple different subspaces, the modal feature sequence F is obtained. temp1 The modal feature sequence F temp1 The data is fed into a feedforward neural network for nonlinear mapping to obtain the modal feature sequence F. temp2 F' is a new feature sequence, F' = {F T ',F A ',F P The formula is as follows:

[0039] F temp1 =MultiHead Self_Attention(M)

[0040] F temp2 =max(0,F temp1 W1+b1)W2+b2

[0041] F′=Layer Norm(F temp1 +F temp2 )

[0042] In the formula, F temp1 Let F represent the initial multimodal feature sequence; M is a sequence containing text, speech, and facial private emotional semantic features; F temp2 F' represents the multimodal feature sequence after fitting the feature distribution of the data; W1 and W2 are linear transformation matrices; b1 and b2 are biases; F' represents the new feature sequence after full fusion of the three modes. T ',F A 'and F P 'respectively F i Text, speech, and facial emotion semantic features in '.

[0043] F' is fed into a fully connected layer, and the feature vectors from the knowledge layer, user layer, and external knowledge layer are then fused using a Contact fusion method to obtain the final vector Z. Z is then input into a multi-head attention mechanism for learning, and a Softmax layer is used for text classification. The resulting sentiment prediction result Z is then output using the following formula: Z = Linear[Concat(F')] T ′,F A ′,F P ′)).

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] 1. This invention addresses the issue of outliers in the collected raw data by designing a data cleaning model based on a combination of adaptive DBSCAN and K-means algorithms. The original dataset is randomly sliced ​​into multiple subsets, and the clustering results are weighted and averaged to obtain the final result. This design reduces recognition time and significantly improves the efficiency and accuracy of subsequent single-modal recognition.

[0046] 2. This invention is based on a mature and efficient single-modal feature extraction method. BERT-BiLSTM, AST, and ResNet-50 pre-trained models are selected for single-modal feature extraction of dialogue text, speech, and facial expressions, ensuring the reliability and accuracy of the extracted modal features.

[0047] 3. This invention constructs an intelligent emotion assessment method based on shared and private information. By enhancing repetitive emotional information and utilizing the uniqueness of private information, it promotes the complementarity of cross-modal emotional information, thereby improving the accuracy of emotion assessment. Furthermore, it employs a bottom-up approach based on a multimodal knowledge graph for real-time emotion assessment. Attached Figure Description

[0048] Figure 1 This is a diagram illustrating the emotion assessment framework based on multimodal knowledge graphs of this invention. Detailed Implementation

[0049] The embodiments of this application are described below, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0050] like Figure 1 As shown, this invention provides an intelligent real-time sentiment assessment method based on multimodal knowledge graphs for social media and online text data, characterized by the following steps:

[0051] S1: Acquire multimodal data from social media and perform data cleaning and preprocessing, wherein the multimodal data includes dialogue text, voice, and facial expressions;

[0052] S2: Based on the characteristics of single-modal data, single-modal feature extraction for dialogue text, speech and facial expressions is carried out using BERT-BiLSTM, AST and ResNet-50 pre-trained models respectively;

[0053] S3: Construct a text-centric multimodal fusion model that considers shared sentiment semantic information among multiple modalities, enhances the expression of key sentiment information within a modality through repeated sentiment information, fully utilizes the unique characteristics of private sentiment semantic information derived from each modality's data, and effectively achieves cross-modal sentiment information complementarity through private sentiment semantic information from different modalities.

[0054] S4: A bottom-up approach is adopted to construct a multimodal knowledge graph. Based on social cognition theory, the DeepFM algorithm is used to analyze user behavior and generate user features, and the K-BERT model is used to capture features of the external knowledge layer.

[0055] S5: Based on the TransformerEncoder structure, it realizes cross-modal emotional information interaction and feature fusion. The input to the Softmax layer completes the recognition of multimodal emotion perception, realizing the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, thereby conducting intelligent real-time emotion assessment.

[0056] In S1, multimodal data from social media is acquired and cleaned and preprocessed, including the following steps: Multimodal experimental data includes conversation text, voice, and facial expressions. This is achieved through the design of automated media client tools for authorization, with chat logs collected in real-time in the background for data processing and analysis. However, the data collected in the background is susceptible to noise and inconsistent data. Low-quality data will lead to low-quality data mining results, affecting subsequent evaluation work.

[0057] In this embodiment, the specific operations are as follows: An adaptive DBSCAN and K-means combined algorithm is used to clean abnormal data in the collected data. Considering the large volume of background data, the clustering time increases with the number of data points. The original dataset is randomly sliced ​​into multiple subsets for computation, and the clustering results are then weighted and averaged to obtain the final result, thereby reducing computation time and improving efficiency. Then, the abnormal data after combined cleaning is removed, and the removed values ​​are filled in using a random forest algorithm. This method efficiently stores data, ensures data quality, and optimizes the data cleaning process, thus achieving efficient cleaning of multi-source heterogeneous data.

[0058] In S2, unimodal feature extraction for dialogue text, speech and facial expressions is performed based on BERT-BiLSTM, AST and ResNet-50 pre-trained models, including the following steps: unimodal feature extraction of dialogue text based on BERT-BiLSTM, unimodal feature extraction of speech based on AST model, and facial expression feature extraction based on ResNet-50.

[0059] In this embodiment, the specific method for feature extraction is as follows: For dialogue text data, BERT-BiLSTM encoding is used to extract contextual information from the utterance; for speech data, AST encoding is used to extract features; for facial expressions, a ResNet-50 pre-trained model is used for feature extraction, as shown in the following formula: X T =BERT-BiLSTM(Text)

[0060] X A =AST(Audio)

[0061] X P =ResNet-50(Picture)

[0062] Among them, X T ∈R l1 ×768, X T X represents the extracted text feature vector sequence. A ∈R l2 ×768, X A X represents the extracted speech feature vector sequence. P ∈R l3 ×2048, X P Let R represent the extracted facial expression feature vector sequence; R is a sequence of real numbers. l1; l2; l3 are the lengths of the text, speech, and facial expression image sequences, respectively.

[0063] In S3, a multimodal fusion model is constructed, which considers the shared sentiment semantic information among multiple modalities. The expression of key sentiment information within a modality is enhanced by its repeated sentiment information. The unique characteristics of private sentiment semantic information originating from each modality's data are fully utilized, and cross-modal sentiment information complementarity is effectively achieved through private sentiment semantic information from different modalities.

[0064] In this embodiment, the specific method for constructing the multimodal fusion model is as follows: The shared sentiment semantic computing module uses the shared sentiment semantic information of the dialogue text signal and the speech signal to enhance the expression of key sentiment information in the text, and calculates the text information sequence T = [t1, t2, ..., t lT The attention weights are wight = {w1, w2, ..., w lT The text information sequences are arranged from largest to smallest, and the semantic sentiment they embody is quantified and expressed as key-value pairs, as shown in the following formula:

[0065]

[0066] Among them, w i Represents the t-th word in the text information sequence i Normalized attention weights for each item; Key represents semantic sentiment key-value pairs;

[0067] Calculate each word pair X in the text sequence A and X P similarity score S T→I ; I∈{A,P}; Use the Softmax function to calculate the attention weights W T→I The shared sentiment semantic feature vector C is obtained by using a cross-modal shared mask matrix. I The features are then concatenated along the text dimension, and a self-attention mechanism layer is used to extract important sentiment information features and enhance key features to obtain F. m Then, the data is fed into an LSTM to add temporal information and obtain the final text sentiment feature F. t The formula is as follows:

[0068] F m =Self_Attention([C A C P ;X T ])

[0069] F t =LSTM(F m )

[0070] Among them, F m Indicates the emotional information characteristics after splicing; C A and C PF represents the shared emotional semantic feature vectors from speech and facial expressions, respectively; t F represents the sentiment features of text with temporal information. t ∈R 1×d d represents the output dimension of the hidden layer of the LSTM network.

[0071] Private sentiment semantic computation focuses on the emotional information within speech and facial expression modalities. It employs an Encoder-Decoder framework, where a pre-trained model encodes and obtains feature sequences, which are then fed into the Decoder module to decode and reconstruct the feature sequences, yielding private sentiment features. A self-attention mechanism layer is used to mine important sentiment features within expressions and speech to increase the weight of key sentiment features. The input to the self-attention mechanism is based on a linear matrix, and facial expressions are integrated using LSTM to obtain a high-level sentiment feature sequence X. I ' Attention And add temporal information to obtain private emotional semantic features F within facial expressions and speech modalities. I The formula is as follows:

[0072]

[0073] F I =LSTM(X) I ' Attention )

[0074] Among them, X I ' Attention Represents a sequence of high-level emotional features in facial expressions and speech; F I Represents private emotional semantic features within facial expressions and speech modalities; Query I 'Feature elements in high-level emotion feature sequences of facial expressions and speech; Key I 'and Value I ' represents the semantic sentiment information key-value pair corresponding to the feature element; Its purpose is to prevent the partial derivatives from being zero when the gradient propagates in the reverse direction.

[0075] In S4, to address the challenges of diverse and heterogeneous data and the difficulty in integrating complementary data in real-time emotion assessment, a method for constructing a multimodal knowledge graph is proposed. This method employs a bottom-up approach, based on social cognition theory, using the DeepFM algorithm to analyze user behavior and generate user features, and relying on the K-BERT model to capture external knowledge layer features, as detailed below:

[0076] The DeepFM algorithm is used to analyze the transformation of continuous features and the learning efficiency problem caused by data sparsity. Continuous variables are discretized and transformed using one-hot encoding. The DeepFM algorithm is used to extract low-order feature combinations, and their correlation is measured by the inner product of features. The DNN model learns high-order feature combinations and shares the embedding layer with DeepFM to enhance feature interaction and further capture user interaction attribute features.

[0077] Knowledge augmentation is achieved by integrating the knowledge graph into the language representation using the K-BERT model to enhance its interpretability. At the knowledge layer, the K-BERT model uses knowledge queries to match entities in the graph and injects triplet information into the text, forming a sentence tree rich in background knowledge. The embedding layer uses the pre-trained language model BERT-BiLSTM to map the semantic information of the sentence tree to a vector space, obtaining token embeddings. The sentence tree uses soft-position embedding, written according to continuous numbering of the main branch and progressive numbering of branches. Segment embeddings are used to label the division of each sentence. The visual layer introduces a visibility matrix to control the connections between words and determine whether they are on the same branch. Finally, the embedding representation and the visibility matrix are fed into a Mask-Transformer encoding layer composed of multiple stacked Mask-self-attention blocks to generate an external knowledge layer feature vector representation containing rich semantic information, achieving a deeper understanding of the text.

[0078] In S5, feature interaction and fusion are performed to conduct intelligent real-time emotion assessment, including the following steps: Based on the Transformer Encoder structure, cross-modal emotional information interaction and feature fusion are realized, input into the Softmax layer to complete the recognition of multimodal emotion perception, and the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers are realized to conduct intelligent real-time emotion assessment.

[0079] In this embodiment, the specific operation is as follows: In the fusion mechanism, the obtained enhanced text features F are... T Voice and facial private emotional semantic features F A and F P This constitutes a new feature sequence M = {F} T ,F A ,F P The modality fusion module is based on a multi-head self-attention mechanism and optimizes the TransformerEncoder structure. After cross-modal information interaction in multiple different subspaces, the modal feature sequence F is obtained. temp1 The modal feature sequence F temp1 The data is fed into a feedforward neural network for nonlinear mapping to obtain the modal feature sequence F.temp2 F' is a new feature sequence, F' = {F' T ,F' A ,F' P The formula is as follows:

[0080] F temp1 =MultiHead Self_Attention(M)

[0081] F temp2 =max(0,F temp1 W1+b1)W2+b2

[0082] F′=Layer Norm(F temp1 +F temp2 )

[0083] In the formula, F temp1 Let F represent the initial multimodal feature sequence; M is a sequence containing text, speech, and facial private emotional semantic features; F temp2 F' represents the multimodal feature sequence after fitting the feature distribution of the data; W1 and W2 are linear transformation matrices; b1 and b2 are biases; F' represents the new feature sequence after full fusion of the three modes. T ',F A 'and F P 'respectively F i Text, speech, and facial emotion semantic features in '.

[0084] F' is fed into a fully connected layer, and the feature vectors from the knowledge layer, user layer, and external knowledge layer are then fused using a Contact fusion method to obtain the final vector Z. Z is then input into a multi-head attention mechanism for learning, and a Softmax layer is used for text classification. The resulting sentiment prediction result Z is then output using the following formula: Z = Linear[Concat(F')] t ′,F a ′,F p ′)]

[0085] This invention presents an intelligent real-time emotion assessment method based on multimodal knowledge graphs for social media and online text data. This method improves the accuracy and real-time performance of emotion assessment, effectively integrating multiple data sources (including dialogue text, voice, facial expressions, and external knowledge layers) to enhance the comprehensive understanding of users' emotional states. The method reduces the complexity of data diversity and cross-modal fusion. Through the construction of a multimodal knowledge graph and the application of a multi-head attention mechanism, it achieves cross-modal interaction and feature fusion of emotional information, enhancing the model's generalization ability and interpretability, thereby improving the accuracy of emotion perception. This method has significant application value in areas such as human resource management, e-commerce customer service, and promoting personalized recommendations.

[0086] The above description of the present invention is provided to enable any person skilled in the art to implement or use the present invention. Various modifications to the present invention will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the scope of protection of the present invention. Therefore, the present invention is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. An intelligent real-time sentiment assessment method based on multimodal knowledge graphs for social media and online text data, characterized in that, Includes the following steps: S1: Acquire multimodal data from social media and perform data cleaning and preprocessing, wherein the multimodal data includes dialogue text, voice, and facial expressions; S2: Based on the characteristics of single-modal data, respectively based on BERT-BiLSTM, AST and ResNet-50 The pre-trained model is designed for single-modal feature extraction of dialogue text, speech, and facial expressions; S3: Construct a text-centric multimodal fusion model that considers shared sentiment semantic information among multiple modalities, enhances the expression of key sentiment information within a modality through repeated sentiment information, fully utilizes the unique characteristics of private sentiment semantic information derived from each modality's data, and effectively achieves cross-modal sentiment information complementarity through private sentiment semantic information from different modalities. S4: Employing a bottom-up approach, a multimodal knowledge graph is constructed based on social cognition theory. DeepFM The algorithm analyzes user behavior and generates user characteristics, relying on K-BERT The model captures features from the external knowledge layer; S5: Based on Transformer Encoder The structure enables cross-modal emotional information interaction and feature fusion, with input... Softmax The layer completes the recognition of multimodal emotion perception, and realizes the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, so as to carry out intelligent real-time emotion assessment. In S1, multimodal data from social media is acquired and cleaned and preprocessed, including the following steps: By designing automated authorization tools for media clients, the backend collects chat logs in real time and performs data processing and analysis; adaptive... DBSCAN and K-means The combined algorithm cleans the outlier data in the collected data; it splits the original dataset into multiple subsets by random slicing, performs calculations on the subsets, and finally calculates the weighted average of the clustering results to obtain the final result; then, the outlier data after combined cleaning is removed and the removed values ​​are filled in using the random forest algorithm; In S3, a multimodal fusion model is constructed, considering the shared sentiment semantic information among multiple modalities. The expression of key sentiment information within a modality is enhanced through repeated sentiment information. The unique characteristics of private sentiment semantic information originating from each modality's data are fully utilized. Cross-modal sentiment information complementarity is effectively achieved through private sentiment semantic information from different modalities. The specific method for constructing the multimodal fusion model is as follows: The shared sentiment semantics computing module uses shared sentiment semantics information from dialogue text signals and speech signals to enhance the expression of key sentiment information in the text; Calculate the text information sequence T =[ t 1, t 2, …, t lT Attention weights wight ={ w 1, w 2, …, w lT The text information sequences are then arranged from largest to smallest; the semantic sentiment embodied in the text information sequences is quantified and expressed in key-value pairs, as shown in the following formula: in, w i Represents the text information sequence number 1 t i Normalized attention weights for each item; Key Represents semantic sentiment key-value pairs; Calculate the speech feature vector sequence for each word in the text sequence. X A and facial expression feature vector sequence X P similarity score S T→I ; I ∈{ A , P },use Softmax The function calculates attention weights and uses a cross-modal shared mask matrix to obtain shared sentiment semantic feature vectors. C I The features are then concatenated along the text dimension, and a self-attention mechanism layer is used to extract important sentiment information features and enhance key features. F m Then send it in LSTM Add time sequence information to obtain the final text sentiment features. F t The formula is as follows: in, X T This represents the extracted text feature vector sequence. F m This indicates the characteristics of the concatenated emotional information. C A and C P These represent shared emotional semantic feature vectors from speech and facial expressions, respectively; F t This indicates the sentiment characteristics of text that contain temporal information. F t ∈ R 1×d , R It is a sequence of real numbers; d for LSTM Network hidden layer output dimension; The private sentiment semantic computing module focuses on the sentiment information within the speech modality and facial expression modality, and adopts... Encoder- Decoder The framework structure, after obtaining the feature sequence through pre-trained model encoding, is fed into... Decoder The module decodes and reassembles feature sequences to obtain private semantic sentiment features; This study utilizes a self-attention mechanism layer to mine important emotional features within facial expressions and speech to increase the weight of key emotional features. The input to the self-attention mechanism is obtained based on a linear matrix. LSTM By integrating facial expressions, a high-level emotional feature sequence is obtained. X I ’ Attention It also adds temporal information to obtain private emotional semantic features within facial expressions and speech modalities. F I The formula is as follows: in, X I ’ Attention This represents a sequence of high-level emotional features in facial expressions and speech. F I Represents private emotional semantic features within facial expressions and speech modalities; Query I ’ Feature elements in high-level emotion feature sequences of facial expressions and speech; Key I ’ and Value I ’ These are the key-value pairs representing semantic sentiment information corresponding to the feature elements; Its purpose is to prevent the partial derivatives of the gradient from being zero when the gradient propagates in the reverse direction; In S4, addressing the challenges of diverse and heterogeneous data and the difficulty in data fusion during real-time emotion assessment, a method for constructing a multimodal knowledge graph is proposed. This method employs a bottom-up approach, based on social cognitive theory, and utilizes... DeepFM Algorithms analyze user behavior to generate user characteristics, relying on K-BERT The model captures features from the external knowledge layer, specifically as follows: use DeepFM The algorithm analyzes the transformation of continuous features and the learning efficiency problem caused by data sparsity, and discretizes the continuous variables. one-hot Encoding conversion, DeepFM The algorithm is used to extract low-order feature combinations, and measures their correlation through the inner product of the features. DNN The model learns high-order feature combinations and... DeepFM shared embedding Embedded layer enhances feature interaction and further captures user interaction attribute features; pass K-BERT The model undergoes knowledge augmentation by integrating knowledge graphs into language representations to improve... K-BERT The interpretability of the model, at the knowledge level, K-BERT The model uses knowledge queries to match entities in the graph and injects triple information into the text to form a sentence tree rich in background knowledge. The embedding layer uses a pre-trained language model. BERT-BiLSTM Mapping the semantic information of the sentence tree to a vector space yields... token Embedding, sentence tree adopts soft-position Embedded, written according to the method of continuous numbering of the main branch and step-by-step numbering of the branches. segment Embeddings are used to label the segmentation of each sentence; the visual layer introduces a visibility matrix to control the relationships between words and determine whether they are on the same branch; finally, the embedding representation and the visibility matrix are passed to multiple... Mask-self-attention Composed of stacked blocks Mask-Transformer The encoding layer generates feature vector representations of the external knowledge layer containing rich semantic information, enabling a deeper understanding of the text; In S5, the aforementioned based on Transformer Encoder The structure enables cross-modal emotional information interaction and feature fusion, with input... Softmax The layer completes the recognition of multimodal emotion perception, realizing the integration and interconnection of dialogue text, voice information, facial expressions and external knowledge layers, thereby conducting intelligent real-time emotion assessment, as detailed below: In the fusion mechanism, the obtained enhanced text features are... F T Voice and facial private emotional semantic features F A and F P This constitutes a new feature sequence. M ={ F T , F A , F P The modality fusion module is based on and optimized using a multi-head self-attention mechanism. Transformer Encoder Structure; after cross-modal information interaction in multiple different subspaces, modal feature sequences are obtained. F temp1 Modal feature sequence F temp1 The data is fed into a feedforward neural network for nonlinear mapping to obtain modal feature sequences. F temp2 , F 'For new feature sequences, F '={ F T ', F A ', F P The formula is as follows: In the formula, F temp1 This represents the initial multimodal feature sequence; M A sequence containing text, speech, and facial private emotional semantic features; F temp2 This represents the multimodal feature sequence after fitting the feature distribution of the data; W 1, W 2 is a linear transformation matrix; b 1, b 2 represents the bias; F 'Represents the new feature sequence after full fusion of the three modes, F T ', F A 'and F P 'Separately F i Text, speech, and facial emotion semantic features in '. Will F 'Send the data into the fully connected layer, and then combine the feature vectors from the knowledge layer, user layer, and external knowledge layer.' Contact The final vector is obtained by fusion. Z ,Will Z The input is fed into a multi-head attention mechanism for learning, employing... Softmax The layer implements text classification, thereby outputting sentiment prediction results. Z The formula is as follows: 。 2. The intelligent real-time sentiment assessment method based on multimodal knowledge graphs for social media and online text data according to claim 1, characterized in that, In S2, based on BERT-BiLSTM , AST and ResNet-50 The extraction of unimodal features from dialogue text, speech, and facial expressions from a pre-trained model includes the following steps: For dialogue text data, use BERT-BiLSTM Encoding extracts contextual information from speech; for speech data, it uses... AST Encoding to extract features, for facial expressions, using ResNet-50 The pre-trained model performs feature extraction using the following formula: in, X T ∈ R l1 ×768, X A ∈ R l2 ×768, X A This represents the extracted speech feature vector sequence. X P ∈ R l3 ×2048, X P This represents the extracted sequence of facial expression feature vectors; l1; l2; l3 These represent the lengths of the text, speech, and facial expression image sequences, respectively.

Images