Emotion recognition method and device, equipment and medium

Abstract

The embodiment of the invention discloses an emotion recognition method and device, equipment and a medium. The method comprises the steps of collecting corresponding multi-modal data for a user, and performing feature extraction on the multi-modal data to determine corresponding discriminative features; constructing a dynamic sub-graph by taking each discriminative feature and the corresponding feature type and confidence as nodes and taking an emotion association relationship among the nodes as a side relationship; wherein the feature type corresponding to the discriminative feature is determined according to the corresponding data type; and matching the dynamic sub-graph with a mode sub-graph in a preset rule base, and determining an emotion recognition result of the user. According to the scheme, user emotion recognition can be realized through graph rule reasoning in a specific application scene, so that the emotion recognition process is explicitly reasonable, and when recognition is wrong, features causing misrecognition can be accurately positioned, and optimization of recognition logic is facilitated.

Description

Technical Field

[0001] This application relates to the field of emotion recognition technology, and in particular to an emotion recognition method, device, equipment and medium. Background Technology

[0002] Emotion recognition is a technology that automatically infers a person's current emotional state by analyzing cues such as facial expressions, tone of voice, body posture, or text content. It combines computer vision, natural language processing, and machine learning to identify basic emotion categories such as happiness, sadness, anger, surprise, fear, and disgust, and can even capture more subtle emotional changes. Emotion recognition has wide applications in various fields, such as improving user experience in human-computer interaction, assessing emotional states in mental health monitoring, analyzing user feedback in customer service, and assisting in emotion management in education. With advancements in deep learning technology, the accuracy and adaptability of emotion recognition are continuously improving, but it also faces challenges such as cross-cultural differences and privacy protection.

[0003] Existing emotion recognition technologies largely rely on a single modality (such as speech only, text only, or facial images only). In complex real-world scenarios, they are susceptible to interference from factors such as missing information, environmental noise, individual differences, and occlusion, resulting in low accuracy and poor robustness. Furthermore, existing deep learning-based multimodal emotion recognition methods, especially those relying on attention mechanisms and complex temporal neural networks (such as Transformers), suffer from a significant "black box" problem. Their decision-making process is uninterpretable, making it difficult to pinpoint which modality's feature extraction is flawed, or whether there are deviations in the fusion or contextual understanding stages when errors occur. This leads to difficulties in debugging, long deployment cycles, and high costs. Moreover, these data-driven models heavily rely on large amounts of labeled data, are sensitive to data bias, and exhibit unstable performance in "long-tail" scenarios not covered by training data. Summary of the Invention

[0004] This application provides an emotion recognition method, apparatus, device, and medium to achieve accurate emotion recognition while making the recognition process explicit, transparent, and reasonable.

[0005] According to one aspect of this application, an emotion recognition method is provided, the method comprising:

[0006] For each user, collect corresponding multimodal data and extract features from the multimodal data to determine the corresponding discriminative features;

[0007] A dynamic subgraph is constructed using each discriminative feature, its corresponding feature type, and confidence level as nodes, and the sentiment relationship between nodes as edge relationships; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type.

[0008] The dynamic subgraph is matched with the pattern subgraph in the preset rule base to determine the user's emotion recognition result.

[0009] According to one aspect of this application, an emotion recognition device is provided, the device comprising:

[0010] The discriminative feature extraction module is used to collect corresponding multimodal data from users and extract features from the multimodal data to determine the corresponding discriminative features.

[0011] The dynamic subgraph construction module is used to construct a dynamic subgraph using each discriminative feature and its corresponding feature type and confidence level as nodes, and the sentiment relationship between each node as edge relationship; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type;

[0012] The emotion recognition result determination module is used to match the dynamic subgraph with the pattern subgraph in the preset rule base to determine the user's emotion recognition result.

[0013] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and,

[0015] A memory that is communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the emotion recognition method of any embodiment of this application.

[0017] According to another aspect of this application, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the emotion recognition method of any embodiment of this application.

[0018] According to another aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the emotion recognition method of any embodiment of this application.

[0019] The technical solution of this application embodiment involves collecting corresponding multimodal data from users and extracting features from the multimodal data to determine corresponding discriminative features; using each discriminative feature and its corresponding feature type and confidence level as nodes, and the emotional association between each node as edge relationships, a dynamic subgraph is constructed; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type; the dynamic subgraph is matched with the pattern subgraph in the preset rule base to determine the user's emotion recognition result. The above solution ensures the comprehensiveness and accuracy of emotion recognition by collecting corresponding multimodal data (such as text, voice, and images) from users and extracting discriminative features for different modalities. Nodes are constructed using discriminative features, their types, and confidence levels, and connected through emotional relationships to form dynamic subgraphs, effectively capturing complex emotional interactions between data. Matching these dynamic subgraphs with pattern subgraphs in a pre-defined rule base achieves accurate identification of user emotions. Furthermore, emotion recognition through graph rules makes each matching and recognition process explicit and transparent, clearly defining the intermediate reasoning to the final emotion recognition result. In the event of an error in emotion recognition, the graph rules can be traced back to identify the feature matching process that led to the misidentification, thereby optimizing the emotion recognition logic.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating an emotion recognition method provided in an embodiment of this application;

[0023] Figure 2 A flowchart illustrating an emotion recognition method provided in another embodiment of this application;

[0024] Figure 3 A flowchart of an emotion recognition method provided in another embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a dynamic sub-graph provided in yet another embodiment of this application;

[0026] Figure 5This is a schematic diagram of the structure of an emotion recognition device provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," "third," "fourth," "actual," "preset," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations. The acquired data is obtained with authorization and will not be disclosed without permission, used for illegal purposes, purposes detrimental to the interests of others, or for personalized analysis or product promotion. It should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary and intended only to illustrate the feasibility of implementing the technical solution of this application, but do not imply that the applicant has already used or necessarily used the relevant content of such solutions.

[0031] Figure 1 This is a flowchart illustrating an emotion recognition method provided in an embodiment of this application. This embodiment is applicable to situations requiring the recognition of a user's emotions. The method can be executed by an emotion recognition device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110. Collect corresponding multimodal data from users, and extract features from the multimodal data to determine the corresponding discriminative features.

[0033] In this context, "user" refers to the individual requiring emotion recognition, and "multimodal data" refers to data collected from different dimensions that reflects the user's emotions. Multimodal data includes at least one of the following: image data, audio data, and text data. Image data is acquired using an image acquisition device whose field of view includes the user; audio data is acquired using a microphone whose audio acquisition range includes the user; and text data can be text input through a terminal or text obtained by converting audio data to text.

[0034] In this embodiment, for the multimodal data collected from the user, feature extraction is performed on each of the multimodal data to determine the corresponding discriminative features. Feature extraction for each multimodal data means that, assuming the multimodal data includes image data, voice data, and text data, feature extraction can be performed on the image data to determine the user's visual features, which can be extracted based on the DeepEmotion model. Feature extraction can be performed on the voice data to determine voice features, which can be extracted based on the SenseVoice model. Feature extraction can be performed on the text data to obtain text features, which can be extracted based on the RoBERTa model. Correspondingly, visual features are the discriminative features corresponding to image data, voice features are the discriminative features corresponding to voice data, and text features are the discriminative features corresponding to text data. The obtained discriminative features are shown in Table 1.

[0035] Table 1

[0036]

[0037] In this embodiment of the application, for the user-collected multimodal data, feature extraction is performed on the multimodal data to determine the corresponding discriminative features, including:

[0038] The user's facial image is captured by an image acquisition device, and visual features are extracted from the facial image.

[0039] The user's voice data is collected through a microphone, and voice features are extracted from the voice data.

[0040] Obtain text data input by the user, and extract text features from the text data to obtain text features;

[0041] Obtain the scene data of the user's location, and extract scene features from the scene data.

[0042] For example, a user's facial image can be captured using an image acquisition device, and visual features can be extracted from the facial image. The user's voice data can be captured using a microphone, and voice features can be extracted from the voice data. The user's input text data can be acquired, and text features can be extracted from the text data. Scene data of the user's context can be acquired, and scene features can be extracted from the scene data. For example, scene features can be extracted based on the contextual relationships between the text data generated by the user; if two users with different voices are conversing, the scene features can be determined to be those of a daily conversation. The above feature extraction process can involve inputting the acquired data into a feature extraction model to obtain the corresponding features. By acquiring multi-source data and extracting multi-source features, joint recognition of multi-source features can be achieved, avoiding the limitations of single features and effectively improving the accuracy of emotion recognition.

[0043] S120. Using each discriminative feature and its corresponding feature type and confidence level as nodes, and the sentiment relationship between each node as edge relationship, construct a dynamic subgraph; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type.

[0044] The feature type corresponding to the discriminative feature is determined based on the corresponding data type. For example, if the discriminative feature corresponds to image data, then its feature type is the visual feature type; if it corresponds to speech data, then its feature type is the speech feature type; and if it corresponds to text data, then its feature type is the text feature type. Confidence is the output confidence level when discriminative features are obtained from feature extraction of multi-source data, reflecting the reliability of the discriminative feature. Emotional association refers to the correlation between emotions reflected by different discriminative features. The most direct correlations include whether the emotions reflected by different discriminative features are the same or different. Additionally, intermediate relationships can be included, such as whether the emotions reflected by different discriminative features are similar, close, all positive, or all negative. This can be adaptively determined based on the required precision of emotion recognition.

[0045] For example, to explicitly represent each discriminative feature and the relationships between them, a dynamic subgraph can be constructed using each discriminative feature, its corresponding feature type, and confidence level as nodes, and the emotional association between nodes as edge relationships. Specifically, each node is a discriminative feature, and it also has a corresponding feature type and confidence level as attributes. Edge relationships between nodes are based on emotional association; if an emotional association exists, they are connected; otherwise, they are not connected. For example, given visual feature N1, speech feature N2, and text feature N3, if N1 and N2 both belong to the emotion "happiness," then N1 and N2 are connected. If N3 belongs to the emotion "disappointment," then N3 is not connected to N1, nor is N3 connected to N2.

[0046] S130. Match the dynamic subgraph with the pattern subgraph in the preset rule base to determine the user's emotion recognition result.

[0047] The preset rule base is a pre-set rule library for emotion recognition based on discriminative features. It contains a large number of rules for emotion recognition based on various discriminative features, defining which discriminative features correspond to which emotions. The pattern subgraph represents the recognition rules corresponding to an emotion, specifying the requirements for discriminative features to be recognized as that emotion. These requirements, defined in the pattern subgraph, include both basic and additional requirements. The basic requirements specify the minimum number of discriminative features required for identification. For example, visual and vocal features must meet certain criteria to be considered an emotion; additional requirements (N3) can further verify the emotion. The basic requirements must be met for an emotion to be identified; additional requirements are optional.

[0048] In this embodiment, a dynamic subgraph can be matched with a pattern subgraph in a preset rule base to determine which pattern subgraph matches the dynamic subgraph, thereby determining the emotion corresponding to that pattern subgraph, which is also the emotion corresponding to the dynamic subgraph. Based on this, the matching results of multiple discriminative features in the dynamic subgraph can be fused to ultimately determine the user's emotion recognition result.

[0049] The technical solution of this application embodiment involves collecting corresponding multimodal data from users and extracting features from the multimodal data to determine corresponding discriminative features; using each discriminative feature and its corresponding feature type and confidence level as nodes, and the emotional association between each node as edge relationships, a dynamic subgraph is constructed; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type; the dynamic subgraph is matched with the pattern subgraph in the preset rule base to determine the user's emotion recognition result. The above solution ensures the comprehensiveness and accuracy of emotion recognition by collecting corresponding multimodal data (such as text, voice, and images) from users and extracting discriminative features for different modalities. Nodes are constructed using discriminative features, their types, and confidence levels, and connected through emotional relationships to form dynamic subgraphs, effectively capturing complex emotional interactions between data. Matching these dynamic subgraphs with pattern subgraphs in a pre-defined rule base achieves accurate identification of user emotions. Furthermore, emotion recognition through graph rules makes each matching and recognition process explicit and transparent, clearly defining the intermediate reasoning to the final emotion recognition result. In the event of an error in emotion recognition, the graph rules can be traced back to identify the feature matching process that led to the misidentification, thereby optimizing the emotion recognition logic.

[0050] Figure 2 The flowchart illustrates another embodiment of an emotion recognition method provided in this application. This application's embodiment is an optimization based on the above embodiment; solutions not described in detail in this application's embodiment are found in the above embodiment. Figure 2 As shown, the method in this embodiment of the application specifically includes the following steps:

[0051] S210. Collect corresponding multimodal data from users and extract features from the multimodal data to determine the corresponding discriminative features.

[0052] S220. Each discriminative feature, its corresponding feature type, and confidence level are used as nodes.

[0053] For example, each discriminative feature can be used as a node, and each discriminative feature is obtained by feature extraction based on the acquired data. In this embodiment of the application, feature extraction for a type of data may yield multiple features. For example, extracting features from image data may yield features such as upturned corners of the mouth and lifted cheeks. Therefore, multiple discriminative features are used as independent features to construct dynamic subgraphs.

[0054] For example, each discriminative feature and its corresponding feature type and confidence level are used as nodes. That is, each discriminative feature is used as a node, and the feature type and confidence level corresponding to the discriminative feature are used as the attributes of the node.

[0055] S230. For any two nodes, based on the discriminative features of the two nodes and the logical relationship between feature entities in the preset rule base relationship layer, determine the emotional association between the two nodes as an edge relationship, and construct a dynamic subgraph.

[0056] For example, for any two nodes, the emotional association between them is determined based on their discriminative features and the logical relationships between feature entities in the preset rule base relationship layer. The preset rule base relationship layer maintains the logical relationships between emotions and feature entities. For instance, the emotion of happiness includes feature entities such as "upturned corners of the mouth" and "raised cheeks," while the emotion of sadness includes feature entities such as "head down" and "expressionless face." In other words, it includes the relationship between emotions and feature entities that reflect those emotions. The logical relationships between feature entities in the preset rule base relationship layer can be queried to determine whether the discriminative features of two nodes belong to the same emotion. If they are associated with the same emotion in the logical relationships between feature entities, the emotional association between the two nodes is determined as an edge relationship.

[0057] In this embodiment of the application, based on the discriminative features of two nodes and the logical relationships between feature entities in the preset rule base relationship layer, the emotional association between the two nodes is determined as an edge relationship, including:

[0058] The discriminative features of two nodes are matched with the logical relationships between feature entities in the preset rule base to determine whether the discriminative features of the two nodes are feature entities associated with the same type of emotion.

[0059] If so, connect the two nodes to form an edge relationship;

[0060] If not, then the two nodes will not be connected.

[0061] In this embodiment, the preset rule base consists of three parts: an entity layer, a relation layer, and a rule layer. The entity layer sets up entities for each emotion category, modal feature entities, and related entities. The relation layer includes the relationships between each emotion category entity and the modal feature entities. The rule layer includes the rules for determining the emotion type based on the modal feature entities.

[0062] The entity layer defines three types of entities, including:

[0063] Emotional entities: discrete emotion categories, such as happy, sad, angry, surprised, neutral;

[0064] Modal features: visual features: such as facial action units AU6 = cheek lift, AU12 = corners of mouth upturn, etc.; speech features: such as prosodic parameters such as fundamental frequency, energy, speech rate, etc.; text features: emotion words such as "happy" and "sad", negation words such as "no" and "not", degree adverbs such as "very" and "slightly".

[0065] Context entity: Application scenario tag, used to dynamically adjust rule weights.

[0066] The relationship layer defines the logical relationships between entities: for example, the emotion of happiness includes the modal feature of upturned corners of the mouth.

[0067] The rule layer stores executable inference rules, with the syntax IF <conditional subgraph> THEN <conclusion> [confidence level].

[0068] For example, in determining the emotional association between two nodes as an edge relationship, the discriminative features of the two nodes can be matched with the logical relationships between feature entities in a preset rule base to determine whether the discriminative features of the two nodes belong to feature entities associated with the same type of emotion. If they are feature entities associated with the same type of emotion, the two nodes are connected to form an edge relationship; otherwise, the two nodes are not connected. An example of the resulting dynamic subgraph is as follows: Figure 3 As shown. N1, N2, N3, N4, N6, and N9 are discriminative features, with the feature types indicated in brackets [ ] and the decimals in parentheses representing confidence levels. N9 is a contextual entity representing a scene feature. Specifically, N1 is a visual feature, represented by AU6 = cheek lift, with a confidence level of 0.88; N2 is a visual feature, represented by AU12 = upturned corners of the mouth, with a confidence level of 0.93; N3 is a speech feature, represented by a 18% increase in fundamental frequency, with a confidence level of 0.85; N4 is a speech feature, represented by a 12% increase in energy, with a confidence level of 0.83; N6 is a text feature, represented by the emotion word "satisfaction," with a confidence level of 0.96; and N9 is a contextual entity, represented by the application scenario being daily conversation, with a confidence level of 0.9. Connected discriminative features are identified as feature entities sharing the same emotion through matching in the relational layer of a preset rule base.

[0069] S240. Match the dynamic subgraph with the pattern subgraph in the preset rule base to determine the user's emotion recognition result.

[0070] The technical solution of this application uses each discriminative feature, its corresponding feature type, and confidence level as nodes. For any two nodes, the emotional association between them is determined as an edge relationship based on the discriminative features of the two nodes and the logical relationship between feature entities in the preset rule base relationship layer. This solution achieves efficient structured modeling of multimodal emotional data by using discriminative features, their feature types, and confidence levels as nodes and dynamically constructing emotional association edges between nodes based on the logical relationship in the preset rule base. It not only accurately captures deep emotional dependencies between features but also improves the accuracy of emotional inference by quantifying node reliability through confidence levels. Furthermore, by leveraging the logical relationship in the preset rule base, it can automatically adapt to emotional interaction patterns in different scenarios, avoiding the limitations of manually defined rules.

[0071] Figure 4 This is a flowchart illustrating an emotion recognition method according to another embodiment of this application. This embodiment is an optimization based on the above embodiments; solutions not described in detail in this embodiment are found in the above embodiments. Figure 4 As shown, the method in this embodiment of the application specifically includes the following steps:

[0072] S310. Collect corresponding multimodal data from users and extract features from the multimodal data to determine the corresponding discriminative features.

[0073] S320. Using each discriminative feature and its corresponding feature type and confidence level as nodes, and the sentiment relationship between each node as edge relationship, construct a dynamic subgraph; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type.

[0074] S330. The dynamic subgraph and the pattern subgraph are matched using a subgraph isomorphic matching algorithm.

[0075] Among them, the VF2 algorithm for subgraph isomorphic matching is an efficient algorithm for graph isomorphic matching, especially suitable for subgraph isomorphic detection problems. Its core idea is to recursively search for node matching relationships to gradually construct the target state. The VF2 algorithm can be used to match dynamic subgraphs with pattern subgraphs, thereby determining the pattern subgraph that matches the dynamic subgraph.

[0076] S340. Determine the user's emotion recognition result based on the emotion type corresponding to the pattern subgraph that matches the dynamic subgraph.

[0077] For example, a pattern subgraph, which is a specific form of the rule layer of the preset rule base, is shown in Table 2. Each pattern subgraph ID corresponds to an emotion. The IF condition specifies the conditions to be met, and the THEN condition specifies the corresponding emotion recognition conclusion. The weights reflect the reliability of the discriminative features in the pattern subgraph in reflecting the emotion type, while the priority reflects the reliability of the pattern subgraph. The requirement of being greater than or equal to a decimal in the IF condition represents the confidence level requirement.

[0078] Table 2

[0079]

[0080] For example, dynamic subgraphs can be matched with pattern subgraphs. The user's recognition result can be determined based on the emotion type corresponding to the pattern subgraph that matches the dynamic subgraph. In other words, during the matching process with Table 2, it is determined which pattern subgraph in Table 2 the discriminative feature matches, and the user's emotion recognition result is determined based on the conclusion emotion corresponding to that pattern subgraph.

[0081] In this embodiment of the application, determining the user's emotion recognition result based on the emotion type corresponding to the pattern subgraph that matches the dynamic subgraph includes:

[0082] If the discriminative features and feature types in the dynamic subgraph match the pattern features and feature types in the pattern subgraph, and the confidence level in the dynamic subgraph meets the confidence level requirement in the pattern subgraph, then the pattern subgraph and the dynamic subgraph are determined to match.

[0083] The emotion type corresponding to the pattern subgraph that matches the dynamic subgraph is used as the candidate emotion type;

[0084] The user's emotion recognition result is determined based on the candidate emotion types.

[0085] For example, during the matching process between dynamic subgraphs and pattern subgraphs, if the discriminative features and feature types of the dynamic subgraph match the pattern features and feature types of the pattern subgraph, and the confidence level of the dynamic subgraph meets the confidence level requirement of the pattern subgraph—for instance, if the discriminative features in the dynamic subgraph are the visual feature AU12 = upturned corners of the mouth (confidence level 0.85) and the textual feature of the emotion word "happy" (confidence level 0.95), matching R1 in Table 2, which means matching the pattern subgraph R1, and the corresponding emotion type is the intermediate emotion "suspected happiness," then "suspected happiness" is selected as the candidate emotion type. If the discriminative feature in the dynamic subgraph is the speech feature of increased fundamental frequency (confidence level 0.85), then based on the already identified candidate emotion type "suspected happiness" and this discriminative feature, matching the pattern subgraph R2 in Table 2, the corresponding emotion type can be determined as the final emotion "happy." If no other discriminative features match other pattern subgraphs in Table 2, then "suspected happiness" and "happy" are selected as candidate emotion types. The user's emotion recognition result is determined based on the candidate emotion type.

[0086] In this embodiment of the application, determining the user's emotion recognition result based on the candidate emotion types includes:

[0087] If the candidate emotion types are the same emotion type, then the candidate emotion type is taken as the emotion recognition result of the user.

[0088] If the candidate emotion types are different emotion types, the user's emotion recognition result is determined according to the weight and priority corresponding to the candidate emotion type; wherein, the priority reflects the credibility of the pattern subgraph, and the weight reflects the credibility of the emotion type reflected by the pattern features in the pattern subgraph.

[0089] For example, if candidate emotion types are associated with the same emotion type, then the candidate emotion type is used as the user's emotion recognition result. If the candidate emotion types are different emotion types, the user's emotion recognition result can be determined based on the weights and priorities corresponding to the candidate emotion types. For instance, if the dynamic subgraph matches R2 and R3 in Table 2, scores are calculated based on the weights and priorities corresponding to R2 and R3 respectively. The weights and priorities can be fused to obtain the score, for example, by multiplying the weights and priorities. The score obtained by multiplying the weights and priorities of R2 is 2.7, and the score obtained by multiplying the weights and priorities of R3 is 1.76. Therefore, the emotion type with the highest score can be selected as the user's emotion recognition result.

[0090] For example, the candidate emotion types "suspected happiness" and "happiness" are both associated with the same emotional tendency, namely the positive emotion "happiness". Since "suspected happiness" is an intermediate emotion, the final emotion "happiness" can be deduced from the intermediate emotion "suspected happiness". Therefore, happiness is taken as the user's emotion recognition result.

[0091] The scheme in this embodiment matches the dynamic subgraph with the pattern subgraph using a subgraph isomorphic matching algorithm; based on the emotion type corresponding to the pattern subgraph that matches the dynamic subgraph, the user's emotion recognition result is determined. This scheme accurately associates the dynamic subgraph and pattern subgraph using the subgraph isomorphic matching algorithm, utilizing graph structure similarity to identify deep mapping relationships between emotion features. This avoids the dependence of traditional methods on single-modality data, significantly improving the accuracy and robustness of emotion recognition. Simultaneously, directly determining the user's emotion based on the emotion type of the matched pattern subgraph simplifies the decision-making process, enabling the system to quickly respond to dynamically changing emotional states. This provides an efficient and interpretable foundation for emotion analysis in applications such as personalized services and mental health monitoring. By matching the dynamic subgraph and pattern subgraph to determine the emotion type recognition result, the emotion recognition process becomes reasonable, presenting the recognition process intuitively, and clarifying the reasoning process while ensuring the accuracy of emotion recognition.

[0092] Figure 5 This is a schematic diagram of the structure of an emotion recognition device provided in an embodiment of this application. This device can execute the emotion recognition method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Figure 5 As shown, the device includes:

[0093] The discriminative feature extraction module 410 is used to collect corresponding multimodal data from users and extract features from the multimodal data to determine the corresponding discriminative features.

[0094] The dynamic subgraph construction module 420 is used to construct a dynamic subgraph using each discriminative feature and its corresponding feature type and confidence level as nodes, and the sentiment relationship between each node as edge relationship; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type;

[0095] The emotion recognition result determination module 430 is used to match the dynamic subgraph with the pattern subgraph in the preset rule base to determine the user's emotion recognition result.

[0096] In this embodiment, the discriminative feature extraction module 410 collects corresponding multimodal data from the user and extracts features from the multimodal data to determine the corresponding discriminative features, including:

[0097] The user's facial image is captured by an image acquisition device, and visual features are extracted from the facial image.

[0098] The user's voice data is collected through a microphone, and voice features are extracted from the voice data.

[0099] Obtain text data input by the user, and extract text features from the text data to obtain text features;

[0100] Obtain the scene data of the user's location, and extract scene features from the scene data.

[0101] In this embodiment, the dynamic subgraph construction module 420 constructs a dynamic subgraph using each discriminative feature and its corresponding feature type and confidence level as nodes, and the emotional association between nodes as edge relationships, including:

[0102] Each discriminative feature, along with its corresponding feature type and confidence level, is used as a node;

[0103] For any two nodes, based on the discriminative features of the two nodes and the logical relationships between feature entities in the pre-defined rule base relationship layer, the emotional association between the two nodes is determined as an edge relationship.

[0104] In this embodiment, the dynamic subgraph construction module 420 determines the emotional association between two nodes as an edge relationship based on the discriminative features of the two nodes and the logical relationship between feature entities in the preset rule base relationship layer, including:

[0105] The discriminative features of two nodes are matched with the logical relationships between feature entities in the preset rule base to determine whether the discriminative features of the two nodes are feature entities associated with the same type of emotion.

[0106] If so, connect the two nodes to form an edge relationship;

[0107] If not, then the two nodes will not be connected.

[0108] In this embodiment, the emotion recognition result determination module 430 matches the dynamic subgraph with the pattern subgraph in the preset rule base to determine the user's emotion recognition result, including:

[0109] The dynamic subgraph and the pattern subgraph are matched using a subgraph isomorphic matching algorithm.

[0110] The user's emotion recognition result is determined based on the emotion type corresponding to the pattern subgraph that matches the dynamic subgraph.

[0111] In this embodiment, the emotion recognition result determination module 430 determines the user's emotion recognition result based on the emotion type corresponding to the pattern subgraph that matches the dynamic subgraph, including:

[0112] If the discriminative features and feature types in the dynamic subgraph match the pattern features and feature types in the pattern subgraph, and the confidence level in the dynamic subgraph meets the confidence level requirement in the pattern subgraph, then the pattern subgraph and the dynamic subgraph are determined to match.

[0113] The emotion type corresponding to the pattern subgraph that matches the dynamic subgraph is used as the candidate emotion type;

[0114] The user's emotion recognition result is determined based on the candidate emotion types.

[0115] In this embodiment of the application, the emotion recognition result determination module 430 determines the user's emotion recognition result based on the candidate emotion types, including:

[0116] If the candidate emotion types are the same emotion type, then the candidate emotion type is taken as the emotion recognition result of the user.

[0117] If the candidate emotion types are different emotion types, the user's emotion recognition result is determined according to the weight and priority corresponding to the candidate emotion type; wherein, the priority reflects the credibility of the pattern subgraph, and the weight reflects the credibility of the emotion type reflected by the pattern features in the pattern subgraph.

[0118] The emotion recognition device provided in this application embodiment can execute an emotion recognition method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0119] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0120] like Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0121] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0122] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as emotion recognition methods.

[0123] In some embodiments, the emotion recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the emotion recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the emotion recognition method by any other suitable means (e.g., by means of firmware).

[0124] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable emotion recognition device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0126] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0128] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0129] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0130] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the emotion recognition method provided in any embodiment of this application.

[0131] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0133] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An emotion recognition method, characterized in that, The method includes: For each user, collect corresponding multimodal data and extract features from the multimodal data to determine the corresponding discriminative features; A dynamic subgraph is constructed using each discriminative feature, its corresponding feature type, and confidence level as nodes, and the sentiment relationship between nodes as edge relationships; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type. The dynamic subgraph is matched with the pattern subgraph in the preset rule base to determine the user's emotion recognition result.

2. The method according to claim 1, characterized in that, For the user-collected multimodal data, feature extraction is performed on each multimodal data to determine the corresponding discriminative features, including: The user's facial image is captured by an image acquisition device, and visual features are extracted from the facial image. The user's voice data is collected through a microphone, and voice features are extracted from the voice data. Obtain text data input by the user, and extract text features from the text data to obtain text features; Obtain the scene data of the user's location, and extract scene features from the scene data.

3. The method according to claim 1, characterized in that, Using each discriminative feature, its corresponding feature type, and confidence level as nodes, and the sentiment relationships between nodes as edge relationships, a dynamic subgraph is constructed, including: Each discriminative feature, along with its corresponding feature type and confidence level, is used as a node; For any two nodes, based on the discriminative features of the two nodes and the logical relationships between feature entities in the pre-defined rule base relationship layer, the emotional association between the two nodes is determined as an edge relationship.

4. The method according to claim 3, characterized in that, Based on the discriminative features of two nodes and the logical relationships between feature entities in the pre-defined rule base relationship layer, the emotional association between the two nodes is determined as an edge relationship, including: The discriminative features of two nodes are matched with the logical relationships between feature entities in the preset rule base to determine whether the discriminative features of the two nodes are feature entities associated with the same type of emotion. If so, connect the two nodes to form an edge relationship; If not, then the two nodes will not be connected.

5. The method according to claim 1, characterized in that, Matching the dynamic subgraph with pattern subgraphs in a preset rule base to determine the user's emotion recognition result includes: The dynamic subgraph and the pattern subgraph are matched using a subgraph isomorphic matching algorithm. The user's emotion recognition result is determined based on the emotion type corresponding to the pattern subgraph that matches the dynamic subgraph.

6. The method according to claim 5, characterized in that, The user's emotion recognition result is determined based on the emotion type corresponding to the pattern subgraph that matches the dynamic subgraph, including: If the discriminative features and feature types in the dynamic subgraph match the pattern features and feature types in the pattern subgraph, and the confidence level in the dynamic subgraph meets the confidence level requirement in the pattern subgraph, then the pattern subgraph and the dynamic subgraph are determined to match. The emotion type corresponding to the pattern subgraph that matches the dynamic subgraph is used as the candidate emotion type; The user's emotion recognition result is determined based on the candidate emotion types.

7. The method according to claim 5, characterized in that, Determining the user's emotion recognition result based on the candidate emotion types includes: If the candidate emotion types are the same emotion type, then the candidate emotion type is taken as the emotion recognition result of the user; If the candidate emotion types are different emotion types, the user's emotion recognition result is determined according to the weight and priority corresponding to the candidate emotion type; wherein, the priority reflects the credibility of the pattern subgraph, and the weight reflects the credibility of the emotion type reflected by the pattern features in the pattern subgraph.

8. An emotion recognition device, characterized in that, The emotion recognition device includes: The discriminative feature extraction module is used to collect corresponding multimodal data from users and extract features from the multimodal data to determine the corresponding discriminative features. The dynamic subgraph construction module is used to construct a dynamic subgraph using each discriminative feature and its corresponding feature type and confidence level as nodes, and the sentiment relationship between each node as edge relationship; wherein, the feature type corresponding to the discriminative feature is determined according to the corresponding data type; The emotion recognition result determination module is used to match the dynamic subgraph with the pattern subgraph in the preset rule base to determine the user's emotion recognition result.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the emotion recognition method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the emotion recognition method according to any one of claims 1-7.

Images