Group emotion recognition method and system based on uncertainty perception learning
By employing an uncertainty-aware learning method, and utilizing Gaussian distribution and uncertainty-sensitive scores to screen key emotional individuals, the problem of poor robustness in group emotion recognition is solved, and efficient recognition is achieved in complex environments.
Patent Information
- Application Number
- CN202311663483.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-06
- Publication Date
- 2024-12-13
AI Technical Summary
Existing group emotion recognition methods cannot effectively handle uncertainty in complex real-world environments, resulting in poor robustness and an inability to adapt to problems such as individual occlusion, uncertainty, and emotional inconsistency within a group, thus affecting recognition performance.
An uncertainty-aware learning approach is adopted, which replaces deterministic feature vectors with probability distribution models, uses Gaussian distribution and uncertainty-sensitive scores to screen key emotional individuals, and enhances robustness by adaptive filtering and weighted fusion of group-level emotional features.
It improves the robustness and accuracy of group emotion recognition, effectively identifies group emotions in complex environments, reduces the impact of uncertainty, and is applicable to more complex real-world scenarios.
Smart Images
Figure CN119131442A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of group emotion recognition in artificial intelligence technology, and in particular to a group emotion recognition method and system based on uncertainty perception learning. Background Art
[0002] Group emotion recognition is a part of human behavior analysis and is a technology that identifies overall emotions in multi-person scenarios. Different from individual-level emotion recognition, group emotion recognition focuses on detecting the emotions expressed by a group of multiple individuals as a whole. This emotion is defined as three types: positive, neutral, and negative. Group emotions are not only affected by individual emotions, but also by personal internal and interpersonal relationships as well as complex real-world scene information. Although individual emotion recognition technology is becoming increasingly mature in fields such as computer vision, speech processing, and multimedia computing, due to the complexity and diversity of the environment in which the group is located, group-level emotion analysis is more challenging than individual-level emotion analysis. Group-level emotion recognition has the following applications in real-world scenarios: 1) Assisting social behavior analysis: analyzing group cohesion, team performance, and students' overall performance in class; 2) Assisting in monitoring public safety: by monitoring video group emotion changes, assisting in abnormal behavior detection, abnormal event detection, and other research areas.
[0003] Current research on group emotion recognition focuses on how to fully exploit multiple emotional clues in the group to learn robust features and how to efficiently aggregate individual features into group features. Among them, the multiple emotional clues involved in the current group emotion recognition methods include: face, object, body, bone information, etc. The mainstream methods mainly focus on two categories of face and object as individual information; the methods of aggregating individuals include: arithmetic operation-based methods (average, majority voting, etc.) and correlation analysis-based methods (recurrent neural networks, graph neural networks, and attention mechanisms, etc.). However, existing methods ignore the inherent uncertainty in unconstrained real environments, such as low-resource samples and occlusions in the group spatial structure, which is attributed to the complexity of group composition and structure. The mutual occlusion between individuals in the group or the occlusion of the individuals themselves seriously leads to the loss of individual information. And the phenomenon of crowding is common in groups, which is closely related to the essential attributes of the group. In addition, different individuals in the same group may express different emotions, but have the same group-level emotional labels, which will confuse the network for emotion prediction. Existing deep learning-based group emotion recognition technologies all represent group-level emotions in a deterministic manner. That is, given an image, they use deep networks to generate deterministic point embeddings, which cannot suppress the negative impact of uncertainty and restrict robust emotional features, resulting in poor group emotion recognition performance. Summary of the invention
[0004] In view of the shortcomings in the prior art, the present invention provides a group emotion recognition method and system based on uncertainty perception learning, which uses probability distribution instead of deterministic feature vectors to model individuals. It can not only generate diversified predictions in the test phase but also use uncertainty sensitivity scores as fusion weights between individuals, thereby enhancing the robustness of group emotion characteristics and thus improving the performance of group emotion recognition.
[0005] The present invention achieves the above technical objectives through the following technical means.
[0006] A group emotion recognition method based on uncertainty-aware learning:
[0007] Perform target detection on the original group samples to obtain individual branches, including face samples and object samples;
[0008] Extract the features corresponding to the individual branches and scene branches respectively;
[0009] The individual features and scene features are respectively fed into the sentiment classification model to obtain the group-level sentiment probability of each branch;
[0010] The group-level sentiment probabilities of each branch are fused by proportional weighting to calculate the group-level sentiment corresponding to the original group sample, which is the final classification result of the group sentiment.
[0011] An uncertainty perception learning module is integrated into the face emotion classification model and the object emotion classification model, and the group-level emotion probability of the individual branch is obtained by the uncertainty perception learning module: the deterministic point embedding of the individual is extended to a random embedding representing different emotion probabilities, and an uncertainty sensitivity score is generated, and the key emotion individuals in the group are screened out by the uncertainty sensitivity score; then, the uncertainty sensitivity score is converted into an importance scalar, and then the weighted group-level features are represented, and the group-level features are sent to the classifier to obtain the group-level emotion probability of the individual branch.
[0012] Furthermore, the individual deterministic point embedding is extended to a random embedding representing different emotion probabilities, specifically, individual features are projected into a Gaussian distribution in a latent space, i.e., parameterized by a mean and a variance.
[0013] Furthermore, the uncertainty sensitive score is obtained by approximating the variance.
[0014] Furthermore, the uncertainty sensitivity score is expressed as:
[0015]
[0016] in: represents the individual (face, object) features after feature encoding, D is the length of the individual feature, σ k，d ,∈k，d Respectively represent σ k and ∈ k The dth component of k is the variance, ∈ k represents random noise.
[0017] Furthermore, the set of key emotional individuals in the group is:
[0018]
[0019] Among them: F K′ To screen out the key emotional individuals in the group based on the uncertainty sensitivity score; Ω = {1, 2, ..., K}, K and K′ are the number of samples before and after screening, thr is the dynamic threshold for dividing the key emotional individual samples, It means that thr is smoothly adjusted by the sigmoid function to control the number of key emotional individual samples, and t is the number of training iterations.
[0020] Furthermore, the group-level features are:
[0021]
[0022] Among them: Importance scalar and Respectively represent the uncertainty sensitivity scores corresponding to the key emotional individuals after screening The maximum and minimum values of ; Random embeddings of different emotions μ k Represents the mean.
[0023] Furthermore, the group-level sentiment probabilities of each branch are fused by proportional weighting to obtain the group-level sentiment, specifically:
[0024] sc group =w f sc f +w o sc o +w s sc s
[0025] Where: sc f ,sc o and sc s denote the group-level emotion probabilities corresponding to the face, object, and scene branches, respectively, and w f 、w o and w s Respectively represent sc f ,sc o and sc s and (scf +sc o +sc s ) ratio, sc group Represents the group-level sentiment prediction results.
[0026] Furthermore, the face emotion classification model and the object emotion classification model are trained in the following manner:
[0027] L=L cls +λ1L reg +λ2L rank +λ3L rec
[0028] Where: L cls represents the group sentiment classification loss, L reg is an additional regularization loss term, L rank is the sorting regularization term, L rec For the reconstruction loss, λ1, λ2, and λ3 are hyperparameters that balance each loss.
[0029] A group emotion recognition system based on uncertainty perception learning, comprising:
[0030] A preprocessing module, used to perform target detection on the original population samples;
[0031] Feature encoder, used to extract features of individual branches and scene branches;
[0032] Sentiment classification model, which inputs individual features and scene features and outputs group-level sentiment probability;
[0033] The proportional weighted fusion module combines the group-level sentiment probabilities of each branch to obtain the group-level sentiment.
[0034] An electronic device comprising a memory and a processor;
[0035] The memory is used to store computer programs;
[0036] The processor is used to execute the computer program and implement the above-mentioned group emotion recognition method based on uncertainty perception learning when executing the computer program.
[0037] A storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the above-mentioned group emotion recognition method based on uncertainty perception learning.
[0038] The beneficial effects of the present invention are:
[0039] (1) The present invention explicitly models the uncertainty of a single sample as a Gaussian random variable, breaking the limitation of traditional methods that can only obtain deterministic point embeddings, alleviating the negative impact of uncertainty, obtaining more robust emotional features, and better generalizing to complex real-world scenarios; secondly, the uncertainty-sensitive scores are adaptively assigned to different individual samples, and the uncertainty-sensitive scores are used as a guide to adaptively filter samples with large uncertainties, screen out key emotional individuals in the group, and perform group-level feature aggregation, thereby suppressing the interference of uncertain individual samples on group emotion recognition. Therefore, the uncertainty-aware learning method has the ability to encode the potential uncertainty between all individuals and suppress the inherent noise in unconstrained real-world scenarios, so that it can simultaneously learn more robust individual and group emotional features to promote classification performance.
[0040] (2) The present invention uses the variance and random noise in the Gaussian distribution to approximate the uncertainty perception score, and adaptively reduces the weight of individuals with high scores (large uncertainty) to aggregate more robust group sentiment features. The complex and changeable environment in which the group is located and the complexity and diversity of the group composition lead to different degrees of uncertainty information in the group. For example, different individuals in the group occlude each other or an individual itself is occluded, resulting in serious loss of individual information; different individuals in the same group have inconsistent emotional expressions, but share a single group emotion label, confusing the emotion classification network. In these cases, if all individuals are treated equally and the damage caused by uncertainty is ignored, the final model performance will decline. The uncertainty-aware learning method measures the importance of different individuals in the group according to their expression forms, obtains the most robust group emotion features, and is therefore suitable for more complex real-world scenarios.
[0041] (3) The uncertainty-aware learning method of the present invention is designed to be lightweight and plug-and-play, and can be easily and efficiently embedded into any sentiment classification model to complete group sentiment recognition tasks. Uncertainty-aware learning-related methods play a key role in expanding the use of emotionally intelligent artificial intelligence agents to applications that require maximum reliability. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a group emotion recognition method and system framework diagram based on uncertainty perception learning according to the present invention;
[0043] Figure 2 This is a flow chart of the uncertainty perception learning algorithm described in the present invention. DETAILED DESCRIPTION
[0044] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.
[0045] like Figure 1As shown, the group emotion recognition system based on uncertainty perception learning of the present invention includes a preprocessing module, a feature encoder, an emotion classification model and a proportional weighted fusion module. The emotion classification model includes a face emotion classification model, an object emotion classification model and a scene emotion classification model, wherein the face emotion classification model and the object emotion classification model are integrated with the uncertainty perception learning module. The preprocessing module performs target detection on the original group sample to obtain the face and object samples of the individual branch. The feature encoder extracts the corresponding emotion features for the individual branch and the scene branch respectively. The individual features and the scene features are respectively sent to the emotion classification model to obtain the group-level emotion probability of each branch. Then, the group-level emotion corresponding to the original group sample is calculated by the proportional weighted fusion module, that is, the final classification result of the group emotion.
[0046] Figure 2 The uncertainty perception learning module represents the process of obtaining the group-level emotion probability of the individual branch: the individual deterministic point embedding is expanded to represent the random embedding of different emotion probabilities, and the uncertainty sensitivity score is generated. The uncertainty sensitivity score is used as a guide to adaptively filter the samples with large uncertainty, screen out the key emotion individuals in the group and assign weights to aggregate the group emotion characteristics; specifically:
[0047] First, the individual features obtained by the feature encoder are sent to the uncertainty encoder. The uncertainty encoder contains a Gaussian sampling operation, which projects the individual features into the Gaussian distribution of the latent space, that is, parameterized by the mean and variance, where the mean represents the feature and the variance represents the uncertainty. The variance is then approximated as an uncertainty sensitivity score, which can measure the importance of each individual in the group. Next, guided by the uncertainty sensitivity score, samples with large uncertainty are adaptively filtered to screen out the key emotional individuals in the group. After that, all individual features are fused into group-level emotional features by using the importance scalar as a weight, and the group-level emotional probability can be obtained.
[0048] use and x s Represents individual features and scene features after feature encoding, according to Figure 1 Process, individual characteristics It needs to be sent to the uncertainty perception learning module for uncertainty modeling to obtain robust group-level sentiment features, jointly optimize sentiment classification loss and uncertainty perception loss, and train the sentiment classification model. In the sentiment classification model testing phase, forward calculation is performed based on the trained sentiment classification model weights, and then the proportional weighted fusion module is used to fuse the group-level sentiment probabilities of the individual and scene sentiment classification models as the final group sentiment prediction result, thus completing the group sentiment recognition of unknown data. The specific process is as follows:
[0049] 1) First, each individual feature (face or object) The representation z in the latent space k Define the mean as u k , the variance is Multivariate Gaussian distribution N:
[0050]
[0051] Among them, the parameters of the Gaussian distribution (mean and variance) are independently mapped by the convolutional neural network: Mean μ k Represents individual characteristics, Represents individual uncertainty.
[0052] At this point, each individual feature is no longer a deterministic point embedding, but is embedded in the latent space from The sampling operation is non-differentiable, however, preventing the back-propagation of gradients during model training. Therefore, a reparameterization trick is used to ensure back-propagation. Specifically, a random noise ∈ is sampled from a normal distribution independent of the model parameters. k ,generate As an equivalent sampling representation, where: ∈ k ~N(0,I), N(0,I) represents a multivariate Gaussian distribution with mean 0 and variance 1.
[0053] 2) By approximating σ n Generate uncertainty sensitivity scores to measure individual uncertainty. The uncertainty sensitivity scores are expressed as:
[0054]
[0055] Where D is the individual characteristic length, σ k，d ,∈ k，d Respectively represent σ k and ∈ k The dth component of . Individuals with larger uncertainties usually correspond to larger uncertainties, and vice versa.
[0056] 3) Guided by the uncertainty sensitivity score, adaptive filtering is performed on samples with large uncertainty, and the key emotional individuals in the group are screened out as follows:
[0057]
[0058] Among them: F K′In order to screen out the key emotional individuals in the group guided by the uncertainty sensitivity score, Ω = {1, 2, ..., K}, K and K′ are the number of samples before and after screening, thr is the dynamic threshold for dividing the key emotional individual samples, It means that thr is smoothly adjusted by the sigmoid function to control the number of key emotional individual samples, and t is the number of training iterations.
[0059] The uncertainty sensitivity scores corresponding to the selected key emotional individuals Converted into importance scalar α k :
[0060]
[0061] in, and Respectively represent the uncertainty sensitivity scores of the key emotional individuals after screening The maximum and minimum values of .
[0062] 4) Finally, the weighted group-level features in the individual branches It can be expressed as:
[0063]
[0064] Among them, K′ is the number of key emotional individual samples; obviously, the importance scalar α k It plays a role similar to the attention mechanism, further weakening the uncertainty of group-level features The interference generated.
[0065] That is, the group-level features of the individual branches are sent to the classifier to minimize the group sentiment classification loss L cls , get the group sentiment probability; group sentiment classification loss L cls It is expressed as:
[0066]
[0067] Where N is the total number of population-level input samples, W c is the cth classifier, C is the number of group sentiment categories, is the parameter of the classifier, y n It is the real group sentiment label.
[0068] 5) Jointly optimize sentiment classification loss and uncertainty perception loss to train the model:
[0069] L=L cls +λ1L reg +λ2L rank +λ3L rec(6)
[0070] Among them, L reg is an additional regularization loss term used to constrain the learning distribution of the standard normal distribution, specifically: L reg =KL(N((z k ;u k , σ k ))||N(0,I)),N((z k ;u k , σ k )) represents the probability distribution space, KL is an indicator to measure the distribution difference; L rank is a ranking regularization term used to normalize the importance scalar α k , clearly constrain the importance of uncertain samples; L rec For a reconstruction loss: The L1 distance between the original individual features and the sampled random features is calculated; λ1, λ2, and λ3 are hyperparameters for balancing each loss. For the scene branch, the cross entropy loss is used during the training phase.
[0071] 6) After the training of the above sentiment classification model integrating the uncertainty perception learning module is completed, the optimal model weight can be obtained. During the test, after completing the group sentiment recognition corresponding to the individual and scene branches of unknown data, the corresponding prediction score is: and Among them sc f ,sc o and sc s represent the group-level emotion probabilities corresponding to individual faces, individual objects, and scene branches, respectively, and C f , C o and C s Represents the corresponding classifier. Through the proportional weighted fusion, the corresponding proportion is used as the weight of each branch to make full use of potential and complementary information. Finally, the final group-level sentiment prediction score is approximately calculated as:
[0072] sc group =w f sc f +w o sc o +w s sc s (7)
[0073] Where W = [w f , w o , w s ] is the ratio of each corresponding branch prediction score to the sum of all branches, sc group Represents the final group-level sentiment prediction result.
[0074] In addition, due to the small size of the GAFF dataset (the source of the original group samples), proportional weighted fusion processes each branch of group emotion recognition separately, which can avoid the phenomenon of reduced fusion performance due to overfitting and obtain better classification performance.
[0075] The uncertainty-aware learning module is an independent module controlled by a set of independent and learnable parameters. The input of the module is the encoded deterministic point embedding of the individual, and the output is the random embedding of the individual and the corresponding importance scalar. The uncertainty-aware learning method can be easily and effectively embedded into any sentiment classification model to achieve group sentiment recognition; the convolutional neural network as the feature encoder can learn robust and reliable sentiment features, promoting the development of the field of group sentiment recognition. Uncertainty-aware learning related methods play a key role in expanding the use of emotionally intelligent artificial intelligence agents to applications that require maximum reliability.
[0076] Based on the same inventive concept as a method for group emotion recognition based on uncertainty perception learning, the present application also provides an electronic device, which includes one or more processors and one or more memories, in which a computer-readable code is stored, wherein the computer-readable code, when executed by one or more processors, implements a method for group emotion recognition based on uncertainty perception learning of the present invention. Among them, the memory may include a non-volatile storage medium and an internal memory; the non-volatile storage medium may store an operating system and a computer-readable code. The computer-readable code includes program instructions, which, when executed, can enable the processor to execute any method for group emotion recognition based on uncertainty perception learning. The processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory provides an environment for the operation of the computer-readable code in the non-volatile storage medium, and when the computer-readable code is executed by the processor, the processor can execute any method for group emotion recognition based on uncertainty perception learning.
[0077] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0078] The computer-readable storage medium may be an internal storage unit of the electronic device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.
[0079] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.
Claims
1. A method for group emotion recognition based on uncertainty perception learning, characterized by: Perform target detection on the original group samples to obtain individual branches, including face samples and object samples; Extract the features corresponding to the individual branches and scene branches respectively; The individual features and scene features are respectively fed into the sentiment classification model to obtain the group-level sentiment probability of each branch; The group-level sentiment probabilities of each branch are fused by proportional weighting to calculate the group-level sentiment corresponding to the original group sample, which is the final classification result of the group sentiment. Uncertainty perception learning modules are integrated into the face emotion classification model and the object emotion classification model, and the group-level emotion probability of the individual branch is obtained by the uncertainty perception learning module: the deterministic point embedding of the individual is extended to a random embedding representing different emotion probabilities, and an uncertainty sensitivity score is generated; the key emotion individuals in the group are screened out by the uncertainty sensitivity score; then, the uncertainty sensitivity score is converted into an importance scalar to further characterize the weighted group-level features, and the group-level features are sent to the classifier to obtain the group-level emotion probability of the individual branch.
2. The group emotion recognition method based on uncertainty perception learning according to claim 1 is characterized in that: The described method extends the deterministic point embedding of individuals to random embeddings representing different emotion probabilities, specifically by projecting individual features into a Gaussian distribution in the latent space, which is parameterized by a mean and a variance.
3. The group emotion recognition method based on uncertainty perception learning according to claim 2 is characterized in that: The uncertainty sensitive score is obtained by approximating the variance.
4. The group emotion recognition method based on uncertainty perception learning according to claim 3 is characterized in that: The uncertainty sensitivity score is expressed as: in: represents the individual feature after feature encoding, D is the length of the individual feature, σ k,d ,∈ k,d Respectively represent σ k and ∈ k The dth component of k is the variance, ∈ k represents random noise; The key emotional individuals in the group are: Among them: F K′ To screen out the key emotional individuals in the group based on the uncertainty sensitivity score; Ω = {1, 2, ..., K}, K and K' are the number of samples before and after screening, thr is the dynamic threshold for dividing the key emotional individual samples, It means that thr is smoothly adjusted by the sigmoid function to control the number of key emotional individual samples, and t is the number of training iterations.
5. The method for group emotion recognition based on uncertainty perception learning according to claim 4 is characterized in that: The population-level features are: Among them: Importance scalar and Respectively represent the uncertainty sensitivity scores corresponding to the key emotional individuals after screening The maximum and minimum values of ; Random embeddings of different emotions μ k Represents the mean.
6. The method for group emotion recognition based on uncertainty perception learning according to claim 5 is characterized in that: The group-level sentiment probabilities of each branch are fused by proportional weighting to obtain the group-level sentiment, specifically: sc group =w f sc f +w o sc o +w s sc s Where: sc f ,sc o and sc s denote the group-level emotion probabilities corresponding to the face, object, and scene branches, respectively, and w f 、w o and w s Respectively represent sc f ,sc o and sc s and (sc f +sc o +sc s ) ratio, sc group Represents the group-level sentiment prediction results.
7. The method for group emotion recognition based on uncertainty perception learning according to claim 1 is characterized in that: The face emotion classification model and the object emotion classification model are trained in the following manner: L=L cls +λ1L reg +λ2L rank +λ3L rec Where: L cls represents the group sentiment classification loss, L reg is an additional regularization loss term, L rank is the sorting regularization term, L rec For the reconstruction loss, λ1, λ2, and λ3 are hyperparameters that balance each loss.
8. A system for implementing the method for group emotion recognition based on uncertainty perception learning as described in any one of claims 1 to 7, characterized in that: include: A preprocessing module, used to perform target detection on the original population samples; Feature encoder, used to extract features of individual branches and scene branches; Sentiment classification model, which inputs individual features and scene features and outputs group-level sentiment probability; The proportional weighted fusion module combines the group-level sentiment probabilities of each branch to obtain the group-level sentiment.
9. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the group emotion recognition method based on uncertainty perception learning as described in any one of claims 1 to 7 when executing the computer program.
10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the group emotion recognition method based on uncertainty perception learning as described in any one of claims 1 to 7.