Multimodal emotion recognition method and apparatus
By integrating a multimodal approach that combines micro-expressions, micro-movements, and text sentiment analysis, the problem of low recognition accuracy caused by a single data source is solved, and more accurate sentiment recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-18
- Publication Date
- 2026-05-22
Smart Images
Figure CN116955601B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, specifically to a multimodal emotion recognition method and apparatus. Background Technology
[0002] Currently, with the development of artificial intelligence, algorithms and technologies have emerged for AI devices to judge human emotions based on facial features, text semantics, and speech. For human psychology and emotion analysis, commonly used visual algorithms include those based on facial micro-expression recognition and micro-expression and brainwave analysis. However, micro-expression-based visual detection algorithms rely solely on facial image analysis, resulting in a single information source and a degree of unreliability. Speech-based emotion analysis techniques, primarily natural language processing and text sentiment analysis (opinion mining, sentiment analysis), are mainly used to analyze emotional tags present in human language. However, this method still suffers from the problem of a single data source, failing to comprehensively collect human emotional data. Regarding knowledge graphs, there is a method that establishes pairing relationships between target images, facial features, micro-expression features, and speech text based on facial attributes to evaluate and predict user behavior. However, this method only analyzes the pairing relationships between facial and text features before evaluating and analyzing user behavior. Therefore, current emotion recognition methods suffer from low accuracy. Summary of the Invention
[0003] This application provides a multimodal emotion recognition method and apparatus to solve the technical problem of low accuracy in emotion recognition.
[0004] In a first aspect, embodiments of this application provide a multimodal emotion recognition method, including:
[0005] Determine the micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vectors of the target user;
[0006] The micro-expression knowledge graph and the micro-action knowledge graph are fused together, and a preset emotion vector is determined based on the fusion result.
[0007] The sentiment analysis result is determined based on the preset sentiment vector and the text sentiment analysis vector.
[0008] The sentiment of the target user is determined based on the sentiment analysis results.
[0009] In one embodiment, fusing the micro-expression knowledge graph with the micro-motion knowledge graph includes:
[0010] Determine the association between the entities in the micro-expression knowledge graph and the micro-action knowledge graph;
[0011] Based on the correlation, the entities in the micro-expression knowledge graph and the micro-action knowledge graph are aligned;
[0012] The micro-expression knowledge graph, after entity alignment, is fused with the micro-action knowledge graph.
[0013] In one embodiment, determining the sentiment analysis result based on the preset sentiment vector and the text sentiment analysis vector includes:
[0014] Determine the correlation between the preset sentiment vector and the text sentiment analysis vector;
[0015] The sentiment analysis results are determined based on the correlation.
[0016] In one embodiment, determining the target user's sentiment based on the sentiment analysis results includes:
[0017] Determine the sentiment probability vector based on the sentiment analysis results;
[0018] The target user's emotion is determined based on the emotion probability vector.
[0019] In one embodiment, before determining the target user's micro-expression knowledge graph, micro-motion knowledge graph, and text sentiment analysis vector, the following steps are included:
[0020] Determine the baseline of the target user;
[0021] Based on the target user's facial video and the baseline, the target user's micro-expressions are identified;
[0022] Based on the micro-expression recognition results, each micro-expression is associated with a preset emotion to obtain the micro-expression knowledge graph.
[0023] In one embodiment, before determining the target user's micro-expression knowledge graph, micro-motion knowledge graph, and text sentiment analysis vector, the method further includes:
[0024] Micro-movements of the target user are identified based on the target user's body video;
[0025] Based on the micro-movement recognition results, each micro-movement is associated with a preset emotion to obtain the micro-movement knowledge graph.
[0026] In one embodiment, before determining the target user's micro-expression knowledge graph, micro-motion knowledge graph, and text sentiment analysis vector, the method further includes:
[0027] The feature vector sequence, target feature vector, and sentiment analysis results of intonation are determined based on the voice signal of the target user.
[0028] The feature vector sequence and the target feature vector are fused to obtain the first feature vector;
[0029] The feature vector sequence is fused with the first feature vector to obtain the second feature vector;
[0030] The target feature vector is fused with the second feature vector to obtain a third feature vector;
[0031] The third feature vector is fused with the sentiment analysis result of the intonation to obtain the text sentiment analysis vector.
[0032] Secondly, embodiments of this application provide a multimodal emotion recognition device, comprising:
[0033] The first determination module is used to determine the target user's micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vector.
[0034] The fusion module is used to fuse the micro-expression knowledge graph and the micro-action knowledge graph, and determine a preset emotion vector based on the fusion result;
[0035] The analysis module is used to determine the sentiment analysis result based on the preset sentiment vector and the text sentiment analysis vector;
[0036] The second determining module is used to determine the target user's sentiment based on the sentiment analysis results.
[0037] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the multimodal emotion recognition method described in the first aspect.
[0038] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal emotion recognition method described in the first aspect.
[0039] The multimodal emotion recognition method and apparatus provided in this application determine the micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vector of the target user. Then, the micro-expression knowledge graph and the micro-action knowledge graph are fused, and a preset emotion vector is determined based on the fusion result. The emotion analysis result is then determined based on the preset emotion vector and the text sentiment analysis vector, and the emotion of the target user is determined based on the emotion analysis result. By constructing a knowledge graph based on multiple modal entities of vision, speech, and text, the emotion recognition result is obtained by integrating multiple modal data, providing more possibilities for emotion judgment, thereby improving the accuracy of emotion recognition. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is one of the flowcharts illustrating the multimodal emotion recognition method provided in the embodiments of this application;
[0042] Figure 2 This is a second schematic flowchart of the multimodal emotion recognition method provided in the embodiments of this application;
[0043] Figure 3 This is the third flowchart illustrating the multimodal emotion recognition method provided in this application embodiment;
[0044] Figure 4 This is the fourth flowchart illustrating the multimodal emotion recognition method provided in this application embodiment;
[0045] Figure 5 This is the fifth flowchart illustrating the multimodal emotion recognition method provided in this application embodiment;
[0046] Figure 6 This is a flowchart illustrating the multimodal emotion recognition method combined with an AR terminal provided in an embodiment of this application.
[0047] Figure 7 This is a schematic diagram of the knowledge graph fusion process provided in the embodiments of this application;
[0048] Figure 8 This is a schematic diagram of the process for generating a micro-expression knowledge graph provided in an embodiment of this application;
[0049] Figure 9 This is a schematic diagram of the process for generating a micro-action knowledge graph provided in an embodiment of this application;
[0050] Figure 10 This is a flowchart illustrating the process of determining the sentiment analysis vector of a text, provided in an embodiment of this application.
[0051] Figure 11 This is a schematic diagram of the structure of the multimodal emotion recognition system provided in the embodiments of this application;
[0052] Figure 12 This is a schematic diagram of the structure of the multimodal emotion recognition device provided in the embodiments of this application;
[0053] Figure 13This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0055] Figure 1 This is one of the flowcharts illustrating the multimodal emotion recognition method provided in this application. (Refer to...) Figure 1 This application provides a multimodal emotion recognition method, which may include:
[0056] Step S10: Determine the target user's micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vector;
[0057] Video information of the target user (e.g., a test subject) is collected, and then the video information is separated into video and audio signals. Based on the video signals, the micro-movements and micro-expressions of the target user are identified. Each micro-expression is associated with a corresponding preset emotion to obtain a micro-expression knowledge graph, and each micro-movement is associated with a corresponding preset emotion to obtain a micro-movement knowledge graph. Simultaneously, tone analysis and text sentiment analysis are performed on the separated audio signals to obtain text sentiment analysis vectors, as described in steps S11 to S110.
[0058] Step S20: The micro-expression knowledge graph and the micro-action knowledge graph are fused together, and a preset emotion vector is determined based on the fusion result;
[0059] After determining the micro-expression knowledge graph and the micro-action knowledge graph, it is necessary to fuse the micro-expression knowledge graph and the micro-action knowledge graph, and determine the preset emotion vector based on the fusion result. For example, determine the correlation between entities in the two knowledge graphs, perform entity mapping based on the correlation, and then perform information fusion within the knowledge graphs. The preset emotion vector is determined based on the fusion result.
[0060] Step S30: Determine the sentiment analysis result based on the preset sentiment vector and the text sentiment analysis vector;
[0061] After determining the preset sentiment vector through knowledge graph fusion, the sentiment analysis result is determined based on the preset sentiment vector and the text sentiment analysis vector. Specifically, the correlation between the preset sentiment vector and the text sentiment analysis vector is determined, and then the sentiment analysis result is determined based on the correlation. For example, in the final sentiment prediction stage, similarity analysis is required, which mainly calculates the correlation between the preset sentiment vector and the text sentiment analysis vector. The correlation measurement indicators can be Euclidean distance, Manhattan distance, Chebyshev distance, cosine angle, etc. It can be understood that the closer the distance, the closer the emotional expression is to consistency; conversely, it can be determined that the subject's words and actions are inconsistent.
[0062] By determining the correlation between the preset sentiment vector and the text sentiment analysis vector, and then determining the sentiment analysis result based on the correlation, it is possible to determine the true sentiment of the target user, thereby improving the accuracy of sentiment recognition.
[0063] Step S40: Determine the target user's sentiment based on the sentiment analysis results.
[0064] After determining the sentiment analysis results, a sentiment probability vector is determined based on these results, and then the sentiment of the target user is determined based on the sentiment probability vector. For example, after data fusion is completed, classification processing is performed through a classification layer, that is, corresponding probability values are generated according to the pre-set number of categories. These probability values are the sentiment probability vector output by inference (all probabilities are summed to 1). The classification methods include, but are not limited to, softmax, SVM, etc., as shown in Table 1, which is a sentiment probability vector table.
[0065] Table 1
[0066] emotion peaceful happy anger disgust surprise … sad pain probability 0.18 0.08 0.12 0.25 0.13 … 0.06 0.11
[0067] Understandably, the higher the probability value, the more authentic the corresponding emotion.
[0068] By determining the target user's emotions through emotion probability vectors, it becomes clear how the target user experiences various emotions, thereby improving the accuracy of emotion recognition.
[0069] This application embodiment determines the target user's micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vector. Then, it fuses the micro-expression knowledge graph and the micro-action knowledge graph, determines a preset sentiment vector based on the fusion result, and then determines the sentiment analysis result based on the preset sentiment vector and the text sentiment analysis vector. Finally, it determines the target user's sentiment based on the sentiment analysis result. By constructing a knowledge graph based on multiple modal entities of vision, speech, and text, it integrates multiple modal data to obtain sentiment recognition results, providing more possibilities for sentiment judgment, thereby improving the accuracy of sentiment recognition.
[0070] refer to Figure 2 , Figure 2 This is a second flowchart illustrating the multimodal emotion recognition method provided in this application embodiment. The step of fusing the micro-expression knowledge graph with the micro-action knowledge graph includes:
[0071] Step S21: Determine the correlation between the entities in the micro-expression knowledge graph and the micro-action knowledge graph;
[0072] Step S22: Align the entities in the micro-expression knowledge graph with those in the micro-action knowledge graph based on the correlation.
[0073] Step S23: The micro-expression knowledge graph after entity alignment is fused with the micro-action knowledge graph.
[0074] It should be noted that studying feature representation and learning methods for different modal information can achieve collaborative representation of multimodal data. If the features of multimodal data (such as language and images) are estimated, their information needs to be projected into a common subspace to achieve collaborative data representation at the knowledge level and support knowledge acquisition based on multimodal fusion.
[0075] The relationships between entities in the micro-expression knowledge graph and the micro-action knowledge graph are determined. Then, based on these relationships, entities in the two graphs are aligned. Finally, the aligned micro-expression knowledge graph and the micro-action knowledge graph are merged. For example, refer to... Figure 7 , Figure 7 This is a schematic diagram of the knowledge graph fusion process provided in this application embodiment. First, two knowledge graphs are initialized. It can be understood that each node in the knowledge graph represents a typical action feature; for example, each node in the micro-expression knowledge graph represents a micro-expression feature, and each node in the micro-action knowledge graph represents a micro-action feature. Since a certain local expression or action of a person usually corresponds to multiple emotions, the nodes between knowledge graphs are not necessarily mapped. Therefore, it is necessary to determine the correlation between the nodes of each knowledge graph. Each node corresponds to at least one entity in the knowledge graph. For example, node A represents the staring action, and its corresponding emotion is anger or fear. Therefore, anger or fear are entities in the knowledge graph. Thus, the correlation between nodes in the knowledge graphs corresponds to the correlation between entities in the knowledge graphs.
[0076] After initializing the knowledge graph, two aggregators continuously update node representations by transmitting information intra-graph and inter-graph. Arrows between two nodes indicate the direction of information aggregation, with thicker lines representing higher attention. The relationships between entities in the two knowledge graphs are determined either manually or through self-learning. Then, entities in the micro-expression knowledge graph and the micro-action knowledge graph are aligned based on these relationships. Finally, information fusion is performed within the knowledge graphs. Figure 7 The brackets [...] indicate concatenation operations. By combining the answer prediction module with the sentiment analysis of the text, a true sentiment analysis result is obtained, and it is determined whether the sentiment expressed in the text and body language are consistent. For example, in... Figure 7 The first step is to determine the micro-expression knowledge graph and the micro-movement knowledge graph. The second step is the aggregation part, which represents entity alignment based on attributes in data fusion to ensure that the two knowledge graphs can be mapped to the same subspace. If there is a one-to-many relationship, the entity with higher similarity is used for entity alignment. For example, if a body movement vector is [anger: 0.8, fear: 0.2] and a facial expression vector is [anger: 0.7, panic: 0.3], then the entities have a strong similarity correlation. If another facial expression vector is [anger: 0.3, pleasure: 0.5, surprise: 0.2], then the entity correlation is not strong. The third step represents information fusion within the knowledge graph. For example, the weights of the entities corresponding to micro-movements and micro-expressions are determined separately. Assuming that the weight of "anger: 0.8" is 0.4 and the weight of "anger: 0.7" is 0.6, then after information fusion, the probability vector of "anger" is: 0.8*0.4 + 0.7*0.6 = 0.74. Among them, knowledge fusion methods include, but are not limited to, distance editing (Levenstein Distance, Wagner and Fisher, EditDistance with Afine Gaps, etc.), set similarity calculation of attribute similarity (Dice coefficient, Jaccard coefficient, TF-IDF, Jaccard coefficient, etc.), entity similarity calculation (aggregation, clustering, representation learning, etc.), and their specific formulas vary depending on the association graph.
[0077] This application embodiment determines the correlation between entities in the micro-expression knowledge graph and the micro-action knowledge graph, aligns the entities in the two knowledge graphs based on the correlation, and then fuses the internal information of the knowledge graphs. In this way, the fused knowledge graph can provide more possibilities for emotion judgment, thereby improving the accuracy of emotion recognition.
[0078] refer to Figure 3 , Figure 3This is a flowchart of the multimodal emotion recognition method provided in this application embodiment, specifically the third step before step S10, which includes:
[0079] Step S11: Determine the baseline of the target user;
[0080] Step S12: Based on the target user's facial video and the baseline, identify the target user's micro-expressions;
[0081] Step S13: Based on the micro-expression recognition results, associate each micro-expression with a preset emotion to obtain the micro-expression knowledge graph.
[0082] Understandably, the human face serves as a medium for conveying emotional information; it acts as an information transmitter. Human facial expressions are primarily divided into macro-expressions and micro-expressions. Macro-expressions are expressions consciously displayed by humans, while micro-expressions are genuine physiological reactions based on human stress. Micro-expressions are brief, subjectively controlled facial expressions that occur unconsciously and can more realistically reflect human emotions. However, micro-expressions typically last only 1 / 25 to 1 / 3 of a second, and the intensity of the movement is very weak, making them difficult to observe and extremely challenging to perform.
[0083] Before collecting facial expressions from target users (such as test subjects), it is necessary to first obtain a baseline of the face in its normal state. In micro-expression analysis, the baseline refers to the fixed position of a person's facial muscles in a normal state. Because the growth and expression of facial muscles vary from person to person, there is no uniform standard for the baseline. The facial baseline can be established using conventional question-and-answer methods or by stimulating the test subject's emotions (such as making them laugh or anger), with the aim of establishing the test subject's facial baseline in their normal state.
[0084] After formally interviewing the participants, the system collects facial videos of them. Then, it performs image preprocessing on the collected facial videos. Specifically, it performs cropping, distortion correction, and facial registration on the faces in the facial videos to process them into standard images that can be processed. Then, the system performs temporal image interpolation on the facial videos to extend the time when micro-expressions appear, making it easier for the system to capture facial micro-expressions. Furthermore, the system amplifies the facial expressions in the facial videos to make the micro-expression features in the images more prominent.
[0085] After image preprocessing, the system extracts facial expression feature points from the face video, decomposes the facial expressions of each part of the face, compares the extracted expressions with the expression feature points in the baseline, calculates the offset of the expression feature points (including the rotation angle of the face), and infers the probability of its emotion based on the preset facial micro-expression feature point offset method, that is, associating each micro-expression with a preset emotion (such as happiness, anger, disgust, surprise, etc.), thereby generating a micro-expression knowledge graph.
[0086] For example, refer to Figure 8 , Figure 8 This is a schematic diagram of the process for generating a micro-expression knowledge graph according to an embodiment of this application. In this embodiment, the steps for generating the micro-expression knowledge graph include: conducting a 1-minute routine interview with the subject, then establishing a baseline for the subject; after establishing the baseline, the system samples the subject's facial expressions and performs image preprocessing on the sampled facial video. The image preprocessing includes facial registration, temporal image interpolation, and amplification of facial expressions; after completing the image preprocessing, facial expression feature points are extracted from the facial video, and then the expression feature parts are decomposed; the extracted expressions are compared with the expression feature points in the baseline to generate expression features for each part; and then the emotion associated with each expression feature is determined, thereby generating a micro-expression knowledge graph.
[0087] This application embodiment determines the baseline of the baseline target user, then identifies the micro-expressions of the baseline target user based on the facial video of the baseline target user and the baseline, and then associates each micro-expression with a preset emotion based on the micro-expression recognition results to obtain a baseline micro-expression knowledge graph. In this way, emotion recognition through the micro-expression knowledge graph can provide more possibilities for emotion judgment, thereby improving the accuracy of emotion recognition.
[0088] refer to Figure 4 , Figure 4 This is a flowchart of the multimodal emotion recognition method provided in this application embodiment, which includes the following steps before step S10:
[0089] Step S14: Identify the micro-movements of the target user based on the target user's body video;
[0090] Step S15: Based on the micro-action recognition results, associate each micro-action with a preset emotion to obtain the micro-action knowledge graph.
[0091] It should be noted that body language is one of the most effective ways for humans to express emotions and is a major source of emotional information in face-to-face communication. While body language can last longer than micro-expressions, it still involves a significant performative element in one-way communication, requiring a high level of expertise from the observer. Micro-expressions are usually subtle movements hidden beneath normal human behavior, such as slight shoulder movements, the angle of the upper body when sitting, the angle of the legs apart, and the direction of the toes.
[0092] Micro-movements are more obvious than micro-expressions and are human stress responses to questions, therefore a baseline is not required. During the questioning of participants, the system collects their body movements. Specifically, the system first detects the participants in the frame, collects body video of the closest participant, then extracts the human movement feature points from the collected body video, combines the interrelated key points, judges each movement, and then associates the possible emotions expressed by the movements, that is, associating each micro-movement with a preset emotion (such as happiness, anger, disgust, surprise, etc.), and finally generates a micro-movement knowledge graph.
[0093] For example, refer to Figure 9 , Figure 9 This is a flowchart illustrating the process of generating a micro-action knowledge graph provided in this embodiment. In this embodiment, the steps for generating the micro-action knowledge graph include: collecting body videos of subjects, extracting human body action feature points from the body videos, decomposing the human body actions, and then performing logical judgments on the action feature points to determine the emotions associated with each micro-action, thereby generating a micro-action knowledge graph.
[0094] This application embodiment identifies the micro-movements of the target user through body video, and then associates each micro-movement with a preset emotion based on the identification results to obtain a micro-movement knowledge graph. In this way, emotion recognition through the micro-movement knowledge graph can provide more possibilities for emotion judgment, thereby improving the accuracy of emotion recognition.
[0095] refer to Figure 5 , Figure 5 This is the fifth flowchart of the multimodal emotion recognition method provided in this application embodiment. Before step S10, the method further includes:
[0096] Step S16: Determine the feature vector sequence, target feature vector, and sentiment analysis results of the tone based on the voice signal of the target user;
[0097] Step S17: Fuse the feature vector sequence and the target feature vector to obtain the first feature vector;
[0098] Step S18: The feature vector sequence is fused with the first feature vector to obtain the second feature vector;
[0099] Step S19: Fuse the target feature vector with the second feature vector to obtain a third feature vector;
[0100] Step S110: The third feature vector is fused with the sentiment analysis result of the tone to obtain the text sentiment analysis vector.
[0101] Understandably, text sentiment analysis, also known as opinion mining or sentiment analysis, is used to uncover people's opinions and emotions. A fundamental step in text sentiment analysis is to classify the polarity of a particular passage of text. This classification can be done at the sentence level or the function level, and the purpose of the classification is to determine whether the opinion expressed in the text is positive, negative, or neutral.
[0102] Text sentiment analysis needs to be combined with tone analysis as a collaborative auxiliary analysis in multimodal analysis. For example, refer to... Figure 10 , Figure 10 This is a flowchart illustrating the process of determining a text sentiment analysis vector according to an embodiment of this application. The text sentiment analysis steps include: collecting speech signals through a terminal device, recognizing and converting the speech signals into text, and then performing text analysis. Specifically, the target language BERT model in the sentiment analysis model is called to encode the text to be processed, obtaining a sequence of feature vectors to represent the contextual relationship between word vectors in the text to be processed. At the same time, specific target words are vectorized to obtain target feature vectors. Then, the feature vector sequence and the target feature vector are fused to obtain a first feature vector. The first feature vector is further fused with the feature vector sequence to obtain a second feature vector. Finally, the second feature vector is fused with the target feature vector to obtain a third feature vector.
[0103] By analyzing the intonation of the speech signal and referring to the text generated after speech recognition, a standard speech audio can be generated. Then, the audio signal and the audio are compared and analyzed to obtain the intonation sentiment analysis result. Finally, the intonation sentiment analysis result is fused with the text analysis result (i.e., the third feature vector) to generate the text sentiment analysis vector.
[0104] This application embodiment performs text sentiment analysis and intonation analysis on the collected speech signals respectively, and then fuses the feature vector sequence obtained from text sentiment analysis with the target feature vector obtained from intonation analysis in a preset manner to obtain a text sentiment analysis vector. In this way, using this text sentiment analysis vector in combination with knowledge graph for sentiment recognition can improve the accuracy of sentiment recognition.
[0105] refer to Figure 6 , Figure 6 This is a flowchart illustrating the multimodal emotion recognition method combined with an AR terminal provided in the embodiments of this application.
[0106] In this embodiment, the user collects video and audio signals from the subject using AR glasses. The AR glasses then send the collected video and audio signals to a processing terminal, such as the cloud or a device (e.g., a mobile phone or computer). The processing terminal decomposes the received video and audio signals to obtain image and audio information. For the image information, the micro-expressions and micro-movements of the subject around the image are extracted to generate micro-expression knowledge graphs and micro-movement knowledge graphs, respectively. For the audio information, the processing terminal converts the audio information into text information, performs intonation analysis, and mines the emotional information in the text to ultimately form textual emotional information.
[0107] After analyzing the multi-dimensional features of audio and video data, the processing terminal uses a multimodal knowledge graph to fuse the generated emotional features, ultimately determining the subject's true emotion (or whether the expressed emotion contradicts the true emotion). After the analysis is complete, the processing terminal sends the results to AR glasses, which then render the results onto the display screen, showing the subject's emotional information next to them.
[0108] In one embodiment, the specific application of multimodal emotion recognition combined with an AR terminal is as follows:
[0109] During interrogation or security screening, the operator collects video and voice information of the subjects by wearing AR glasses. At the beginning, the operator will ask the subjects a routine question for 1 minute. The system will obtain the subject's baseline within 1 minute. When the operator formally asks the subject questions on the target topic, the system will collect and analyze the subject's micro-expressions, micro-movements, and voice data. The system background uses various AI technologies such as knowledge graphs to analyze and judge the subject's true emotions. Finally, the judgment result is rendered on the operator's AR glasses, thus realizing a complete closed loop of data collection, processing and feedback.
[0110] This application's embodiments introduce a multimodal knowledge graph to achieve analysis of multi-source data, enabling a more objective, scientific, and effective judgment of people's true emotions. Secondly, this embodiment demonstrates how to associate micro-expressions and micro-movements with the knowledge graph, distinguishing it from traditional neural networks and obtaining more diverse and multi-dimensional emotional information, providing more possibilities for emotion judgment, thereby improving the accuracy of emotion recognition.
[0111] In another embodiment, reference Figure 11 , Figure 11 This is a schematic diagram of the structure of a multimodal emotion recognition system provided in this application embodiment. This application embodiment provides a multimodal emotion recognition system, which mainly consists of six modules: storage, container cluster management, model inference service, algorithm backend service, gateway routing, and frontend display.
[0112] The front-end display can consist of various devices, such as desktop devices or web-based devices, therefore... Figure 11 It is not displayed in the text.
[0113] The underlying storage infrastructure includes, but is not limited to, cephfs, HDFS, NFS, etc., which mainly provide underlying storage services for the entire system.
[0114] Container cluster management primarily uses Kubernetes to manage containerized applications across multiple hosts on a cloud platform. It provides a mechanism for application deployment, planning, updating, and maintenance, aiming to make containerized applications simple and efficient. Kubernetes, as a component for algorithm container management, plays a role in deploying, planning, updating, and maintaining algorithm inference services within the system. Components with similar functions include, but are not limited to, Kubernetes, MLFlow, and Clipper.
[0115] The containerized service primarily uses Docker or containerd as the underlying container service to provide services externally. These containers provide file storage services, database services, vector search engine services, and algorithm inference services. File storage components include, but are not limited to, minio and FastDFS; database components include, but are not limited to, MySQL, MariaDB, and HBase; vector search engines include, but are not limited to, Milvus and ElasticSearch; and algorithm model inference components include, but are not limited to, TensorFlow Serving, NVIDIA Triton, and KFServing. The model inference service is mainly used to launch various algorithms, receive input data, and output the processed results.
[0116] The algorithm service backend mainly comprises two parts: the algorithm service backend and the system backend. The algorithm backend primarily receives metadata (such as images and audio) and intermediate data from the system backend (such as intermediate processing results returned by the model inference service, human pose points, facial key points, etc.). Its main function is to coordinate the data transfer between the algorithm inference service and to provide logical services beyond algorithm inference and prediction, completing some pre- and post-processing of data for algorithm inference services. The system backend, on the other hand, simply transmits metadata to the algorithm backend and receives the processed results. The system backend is primarily responsible for the overall system operation, handling interactions between the frontend and backend, and managing the system's various backend components.
[0117] The gateway routing part mainly connects to the system backend, exposes external interfaces, and enables interaction with the frontend (end devices or web devices).
[0118] In this application example, the algorithm is implemented by calling a containerized algorithm (model) inference service from the algorithm service backend. The specific implementation steps include: After the system receives video input, it first performs audio-video separation, separating the video signal into image and audio signals. The audio signal undergoes speech-to-text extraction, Chinese preprocessing, text feature extraction, and classification model processing to generate a speech-related graph neural network. The image information undergoes object detection and human pose point extraction, and utilizes the correlation information between human pose points and the temporal correlation information of human pose points to generate a pose point-related graph neural network. Using the information extracted from image object detection, face location and key points are extracted, generating a face key point-related graph neural network. Finally, graph neural network fusion technology is used to fuse the generated graph neural networks to obtain the final sentiment analysis result.
[0119] The multimodal emotion recognition system provided in this application constructs a knowledge graph based on multiple modal entities, including visual, speech, and text, and integrates multiple modal data to obtain emotion recognition results, providing more possibilities for emotion judgment and thus improving the accuracy of emotion recognition.
[0120] The multimodal emotion recognition device provided in the embodiments of this application will be described below. The multimodal emotion recognition device described below can be referred to in correspondence with the multimodal emotion recognition method described above.
[0121] refer to Figure 12 , Figure 12 This is a schematic diagram of the structure of the multimodal emotion recognition device provided in the embodiments of this application. The multimodal emotion recognition device provided in the embodiments of this application includes a first determining module 1210, a fusion module 1220, an analysis module 1230, and a second determining module 1240.
[0122] The first determining module 1210 is used to determine the target user's micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vector;
[0123] The fusion module 1220 is used to fuse the micro-expression knowledge graph and the micro-action knowledge graph, and determine a preset emotion vector based on the fusion result;
[0124] The analysis module 1230 is used to determine the sentiment analysis result based on the preset sentiment vector and the text sentiment analysis vector.
[0125] The second determining module 1240 is used to determine the target user's sentiment based on the sentiment analysis results.
[0126] This application embodiment determines the target user's micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vector. Then, it fuses the micro-expression knowledge graph and the micro-action knowledge graph, determines a preset sentiment vector based on the fusion result, and then determines the sentiment analysis result based on the preset sentiment vector and the text sentiment analysis vector. Finally, it determines the target user's sentiment based on the sentiment analysis result. By constructing a knowledge graph based on multiple modal entities of vision, speech, and text, it integrates multiple modal data to obtain sentiment recognition results, providing more possibilities for sentiment judgment, thereby improving the accuracy of sentiment recognition.
[0127] In one embodiment, the fusion module 1220 is specifically used for:
[0128] Determine the association between the entities in the micro-expression knowledge graph and the micro-action knowledge graph;
[0129] Based on the correlation, the entities in the micro-expression knowledge graph and the micro-action knowledge graph are aligned;
[0130] The micro-expression knowledge graph, after entity alignment, is fused with the micro-action knowledge graph.
[0131] In one embodiment, the analysis module 1230 is specifically used for:
[0132] Determine the correlation between the preset sentiment vector and the text sentiment analysis vector;
[0133] The sentiment analysis results are determined based on the correlation.
[0134] In one embodiment, the second determining module 1240 is specifically used for:
[0135] Determine the sentiment probability vector based on the sentiment analysis results;
[0136] The target user's emotion is determined based on the emotion probability vector.
[0137] In one embodiment, the first determining module 1210 is specifically used for:
[0138] Determine the baseline of the target user;
[0139] Based on the target user's facial video and the baseline, the target user's micro-expressions are identified;
[0140] Based on the micro-expression recognition results, each micro-expression is associated with a preset emotion to obtain the micro-expression knowledge graph.
[0141] In one embodiment, the first determining module 1210 is specifically used for:
[0142] Micro-movements of the target user are identified based on the target user's body video;
[0143] Based on the micro-movement recognition results, each micro-movement is associated with a preset emotion to obtain the micro-movement knowledge graph.
[0144] In one embodiment, the first determining module 1210 is specifically used for:
[0145] The feature vector sequence, target feature vector, and sentiment analysis results of intonation are determined based on the voice signal of the target user.
[0146] The feature vector sequence and the target feature vector are fused to obtain the first feature vector;
[0147] The feature vector sequence is fused with the first feature vector to obtain the second feature vector;
[0148] The target feature vector is fused with the second feature vector to obtain a third feature vector;
[0149] The third feature vector is fused with the sentiment analysis result of the intonation to obtain the text sentiment analysis vector.
[0150] Figure 13 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 13As shown, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330, and a communication bus 1340, wherein the processor 1310, the communication interface 1320, and the memory 1330 communicate with each other via the communication bus 1340. The processor 1310 can call a computer program in the memory 1330 to execute the steps of a multimodal emotion recognition method, such as including:
[0151] Determine the micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vectors of the target user;
[0152] The micro-expression knowledge graph and the micro-action knowledge graph are fused together, and a preset emotion vector is determined based on the fusion result.
[0153] The sentiment analysis result is determined based on the preset sentiment vector and the text sentiment analysis vector.
[0154] The sentiment of the target user is determined based on the sentiment analysis results.
[0155] Furthermore, the logical instructions in the aforementioned memory 1330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0156] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the multimodal emotion recognition method provided in the above embodiments, such as including:
[0157] Determine the micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vectors of the target user;
[0158] The micro-expression knowledge graph and the micro-action knowledge graph are fused together, and a preset emotion vector is determined based on the fusion result.
[0159] The sentiment analysis result is determined based on the preset sentiment vector and the text sentiment analysis vector.
[0160] The sentiment of the target user is determined based on the sentiment analysis results.
[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A multimodal emotion recognition method, characterized in that, include: Determine the micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vectors of the target user; The micro-expression knowledge graph and the micro-action knowledge graph are fused together, and a preset emotion vector is determined based on the fusion result. The sentiment analysis result is determined based on the preset sentiment vector and the text sentiment analysis vector. The sentiment of the target user is determined based on the sentiment analysis results; The micro-expression knowledge graph was determined based on the following method: Determine the baseline of the target user; Based on the target user's facial video and the baseline, the target user's micro-expressions are identified; Based on the micro-expression recognition results, each micro-expression is associated with a preset emotion to obtain the micro-expression knowledge graph; The micro-action knowledge graph was determined based on the following method: Micro-movements of the target user are identified based on the target user's body video; Based on the micro-action recognition results, each micro-action is associated with a preset emotion to obtain the micro-action knowledge graph; The process of fusing the micro-expression knowledge graph with the micro-motion knowledge graph includes: Determine the association between the entities in the micro-expression knowledge graph and the micro-action knowledge graph; Based on the correlation, the entities in the micro-expression knowledge graph and the micro-action knowledge graph are aligned; The micro-expression knowledge graph, after entity alignment, is fused with the micro-action knowledge graph.
2. The multimodal emotion recognition method according to claim 1, characterized in that, The step of determining the sentiment analysis result based on the preset sentiment vector and the text sentiment analysis vector includes: Determine the correlation between the preset sentiment vector and the text sentiment analysis vector; The sentiment analysis results are determined based on the correlation.
3. The multimodal emotion recognition method according to claim 1, characterized in that, Determining the target user's sentiment based on the sentiment analysis results includes: Determine the sentiment probability vector based on the sentiment analysis results; The target user's emotion is determined based on the emotion probability vector.
4. The multimodal emotion recognition method according to claim 1, characterized in that, The text sentiment analysis vector is determined based on the following method: The feature vector sequence, target feature vector, and sentiment analysis results of intonation are determined based on the voice signal of the target user. The feature vector sequence and the target feature vector are fused to obtain the first feature vector; The feature vector sequence is fused with the first feature vector to obtain the second feature vector; The target feature vector is fused with the second feature vector to obtain a third feature vector; The third feature vector is fused with the sentiment analysis result of the intonation to obtain the text sentiment analysis vector.
5. A multimodal emotion recognition device, characterized in that, include: The first determination module is used to determine the target user's micro-expression knowledge graph, micro-action knowledge graph, and text sentiment analysis vector. The fusion module is used to fuse the micro-expression knowledge graph and the micro-action knowledge graph, and determine a preset emotion vector based on the fusion result; The analysis module is used to determine the sentiment analysis result based on the preset sentiment vector and the text sentiment analysis vector; The second determining module is used to determine the sentiment of the target user based on the sentiment analysis results; The first determining module is used to determine the baseline of the target user; Based on the target user's facial video and the baseline, the target user's micro-expressions are identified; based on the micro-expression identification results, each micro-expression is associated with a preset emotion to obtain the micro-expression knowledge graph; The first determining module is used to identify the micro-movements of the target user based on the target user's body video; and to associate each micro-movement with a preset emotion based on the micro-movement identification results to obtain the micro-movement knowledge graph. The fusion module is further configured to determine the correlation between each entity in the micro-expression knowledge graph and the micro-motion knowledge graph; and to align the entities in the micro-expression knowledge graph and the micro-motion knowledge graph according to the correlation. The micro-expression knowledge graph, after entity alignment, is fused with the micro-action knowledge graph.
6. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal emotion recognition method according to any one of claims 1 to 4.
7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimodal emotion recognition method according to any one of claims 1 to 4.
Citation Information
Patent Citations
Multi-round dialogue method and system integrating knowledge graph and emotion supervision
CN111651609A
User psychological state recognition method and system
CN112052869A