An emotion recognition method and system considering environmental perception

Abstract

The present application relates to a kind of emotional recognition method and system considering environmental perception, method includes: the physiological data and environmental data of target object are collected;The collected physiological data and environmental data are preprocessed;The physiological data after pre-processing is unsupervised contrast learning pre-training, and pre-training physiological encoder is obtained;The physiological data after pre-processing is input into pre-training physiological encoder, and physiological representation is extracted;The environmental data after pre-processing is input into environmental perception representation extraction module, and environmental perception representation is extracted;Physiological representation and environmental perception representation are fused, and joint representation is obtained;Joint representation is input into emotional recognition module, and emotional recognition result is obtained.Compared with prior art, by introducing environmental perception representation, environmental information is involved in emotional recognition in a manner more consistent with actual emotion formation mechanism;At the same time, the robustness of physiological representation is improved by unsupervised contrast learning, so as to improve the accuracy and applicability of emotional recognition.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and in particular to an emotion recognition method and system that takes into account environmental perception. Background Technology

[0002] Emotion recognition technology is widely used in scenarios such as intelligent human-computer interaction, health monitoring, smart offices, smart cockpits, and smart buildings. Existing emotion recognition methods typically build recognition models based on physiological signals such as electrocardiogram (ECG), electrodermal conductivity (EDC), and electroencephalogram (EEG). While physiological signals can reflect emotion-related autonomic nervous activity to some extent, their expression is easily affected by individual differences and changes in environmental conditions, thus affecting the recognition stability of the model in real-world applications.

[0003] On the other hand, the influence of environmental conditions on emotions is not only through the direct effect of objective environmental variables, but is also moderated by the individual's subjective environmental perception. Even under the same environmental conditions, different individuals may exhibit different environmental perceptions, and these differences in environmental perception further affect their emotional experience.

[0004] For example, the invention disclosed in CN116343284A, which discloses a multi-feature outdoor environment emotion recognition method based on an attention mechanism, relies heavily on environmental information as ordinary auxiliary features directly input into the emotion recognition model. It lacks modeling of thermal perception as an intermediate semantic layer, making it difficult to depict the chain of interaction between "environment-thermal perception-emotion." Furthermore, directly relying on supervised learning to extract physiological features often requires a large number of labeled samples, and these features are easily affected by noise and individual differences.

[0005] Therefore, it is necessary to propose an emotion recognition scheme with a simpler structure and clearer engineering implementation. This scheme extracts environmental perception representations from environmental data and integrates them with physiological representations obtained through unsupervised comparative learning, thereby improving the emotion recognition effect. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art, which mainly uses environmental information as ordinary auxiliary features and directly inputs it into the emotion recognition model, and to provide an emotion recognition method and system that takes environmental perception into account, with a simpler structure and clearer engineering implementation.

[0007] The objective of this invention can be achieved through the following technical solutions: An emotion recognition method that takes into account environmental perception includes the following steps: Collect physiological data of the target object and environmental data of the environment in which the target object is located; Preprocess the collected physiological and environmental data; Unsupervised contrastive learning pre-training is performed on the preprocessed physiological data to obtain a pre-trained physiological encoder; The preprocessed physiological data is input into the pre-trained physiological encoder to extract physiological representations; The preprocessed environmental data is input into the pre-built and trained environmental perception representation extraction module to extract the environmental perception representation; The physiological and environmental perception representations are fused to obtain a joint representation. The joint representation is input into a pre-built and trained emotion recognition module to obtain the emotion recognition result.

[0008] Furthermore, the physiological data includes one or more of the following: electrocardiogram (ECG) signals, acceleration signals, skin conductance signals, respiratory signals, and electroencephalogram (EEG) signals.

[0009] Furthermore, the environmental data includes at least temperature data and humidity data, and further includes one or more of illuminance, correlated color temperature, particulate matter concentration, carbon dioxide concentration, and noise.

[0010] Furthermore, the unsupervised contrastive learning pre-training of the preprocessed physiological data includes: Data augmentation is performed on the same preprocessed physiological data sample to construct at least two augmented views; Each enhanced view is input into a shared physiological coding network to obtain the corresponding physiological representation; Each physiological representation is input into the projection head to obtain feature representations in the contrastive learning space, and the consistency of the feature representations is constrained by contrastive loss, thereby pre-training the physiological coding network.

[0011] Furthermore, the contrast loss is based on an enhanced view derived from the same physiological data sample. and As positive sample pairs Enhanced view of the remaining physiological data samples As negative sample pairs This allows for the calculation of the contrast loss; For any positive sample pair The corresponding expression for calculating the contrast loss is: In the formula, Positive sample pairs The comparative loss, For similarity function, To enhance the view Feature representation, For temperature coefficient, N This represents the total number of samples within the batch.

[0012] Furthermore, the data augmentation methods include one or more of the following: time segment rearrangement, random pruning, noise perturbation, amplitude scaling, and time masking.

[0013] Furthermore, the environmental perception representation is a thermal perception representation or a light perception representation.

[0014] Furthermore, the thermal perception representation is an intermediate feature vector of the environmental data in the latent space of the environmental perception representation extraction module, used to represent the target object's subjective perception tendency of hot and cold in the current environmental state. The environmental perception representation extraction module is a multilayer perceptron model; During the training process of the environmental perception representation extraction module, a thermal perception prediction head is added to predict the thermal perception state based on the thermal perception representation output by the environmental perception representation extraction module, and compare it with the corresponding label to calculate the loss function to optimize the parameters of the environmental perception representation extraction module.

[0015] Furthermore, the fusion method of the physiological representation and the environmental perception representation is any one of weighted fusion, gated fusion, and attention fusion; The emotion recognition results include one or more of the following: arousal, valence, discrete emotion category, and continuous emotion value.

[0016] The present invention also provides an emotion recognition system that implements the emotion recognition method considering environmental perception as described above, comprising: The data acquisition module is used to collect physiological data of the target object and environmental data of the environment in which the target object is located; The preprocessing module is used to preprocess the collected physiological and environmental data; The pre-training module is used to perform unsupervised contrastive learning pre-training on preprocessed physiological data to obtain a pre-trained physiological encoder. The physiological characterization extraction module is used to input the pre-processed physiological data into the pre-trained physiological encoder to extract physiological characterization. The environmental perception representation extraction module is used to input preprocessed environmental data into the pre-built and trained environmental perception representation extraction module to extract environmental perception representations. The fusion module is used to fuse the physiological representation and the environmental perception representation to obtain a joint representation; An emotion recognition module is used to input the joint representation into a pre-built and trained emotion recognition module to obtain emotion recognition results.

[0017] Compared with the prior art, the present invention has the following advantages: (1) This invention does not simply splice the original environmental variables into the emotion classifier. Instead, it extracts environmental perception representations from environmental data to represent the subjective perception tendency of the target object under the current environmental conditions of hot or cold or light and dark. Then, it integrates the environmental perception representations with the physiological representations extracted based on physiological data, so that the environmental information enters the recognition process in a way that is more in line with the emotion formation mechanism, and obtains more accurate and stable emotion recognition results.

[0018] (2) The present invention takes into account that comparative learning of physiological signals helps to obtain more robust representations, thereby extracting representations from physiological signals through unsupervised comparative learning, which can improve feature robustness and discriminability while reducing label dependence.

[0019] (3) The present invention is simple to implement and easy to deploy in wearable devices, edge intelligent devices and intelligent environmental control systems. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an emotion recognition method that takes into account environmental perception, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of an emotion recognition network structure that takes into account heat perception, provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0024] Example 1 like Figure 1 As shown, this embodiment provides an emotion recognition method that takes into account environmental perception, including the following steps: S1: Collect physiological data of the target object and environmental data of the environment in which the target object is located; Physiological data include one or more of the following: electrocardiogram (ECG) signals, acceleration signals, skin conductance signals, respiratory signals, and electroencephalogram (EEG) signals.

[0025] Environmental data includes at least temperature and humidity data, and further includes one or more of illuminance, correlated color temperature, particulate matter concentration, carbon dioxide concentration, and noise.

[0026] S2: Preprocess the collected physiological and environmental data; For physiological data, preprocessing options may include filtering, denoising, artifact correction, outlier removal, windowing, and normalization. For environmental data, preprocessing options may include time synchronization, missing value imputation, sliding window aggregation, and normalization.

[0027] S3: Perform unsupervised contrastive learning pre-training on the preprocessed physiological data to obtain a pre-trained physiological encoder; Among them, such as Figure 2 As shown, unsupervised contrastive learning pre-training includes: Data augmentation is performed on the same preprocessed physiological data sample to construct at least two augmented views; Each enhanced view is input into a shared physiological coding network to obtain the corresponding physiological representation; Each physiological representation is input into the projection head to obtain feature representations in the contrastive learning space, and the consistency of the feature representations is constrained by contrastive loss, thereby pre-training the physiological coding network.

[0028] The pre-trained physiological encoder is transferred to the classification training stage.

[0029] Data augmentation methods include one or more of the following: time slice rearrangement, random pruning, noise perturbation, amplitude scaling, and time masking.

[0030] S4: Input the preprocessed physiological data into the pre-trained physiological encoder to extract physiological representations; S5: Input the preprocessed environmental data into the pre-built and trained environmental perception representation extraction module to extract the environmental perception representation; Environmental perception is represented as either thermal perception or light perception.

[0031] Among them, thermal perception representation is the intermediate feature vector of environmental data in the latent space of the environmental perception representation extraction module, which is used to represent the target object's subjective perception tendency of hot and cold in the current environmental state. The environmental perception representation extraction module is a multilayer perceptron model; During the training process of the environmental perception representation extraction module, a thermal perception prediction head is added to predict the thermal perception state based on the thermal perception representation output by the environmental perception representation extraction module, and compare it with the corresponding label to calculate the loss function to optimize the parameters of the environmental perception representation extraction module.

[0032] S6: The physiological representation and the environmental perception representation are fused to obtain a joint representation; The fusion method between physiological representation and environmental perception representation can be any one of weighted fusion, gated fusion, and attention fusion; S7: Input the joint representation into the pre-built and trained emotion recognition module to obtain the emotion recognition result; Emotion recognition results include one or more of the following: arousal, valence, discrete emotion category, and continuous emotion value.

[0033] In this embodiment, emotion recognition in an office setting is used as an example. In this scenario, the target is primarily in an indoor office environment, and their emotional state is represented using two dimensions: arousal and valence. Corresponding binary classification tasks are then constructed for each dimension. Specifically, environmental perception information in this embodiment is represented using thermal perception, which is quantified using a seven-level scale to describe the target's subjective perception of the current thermal environment.

[0034] Correspondingly, the processing steps of the above method include: Step 1: Collect physiological and environmental data of the target object.

[0035] Physiological data of the target object is collected through wearable devices, and environmental data of the target object's environment is collected through environmental sensors.

[0036] In this embodiment, physiological data includes electrocardiogram (ECG) signals; environmental data includes ambient temperature, relative humidity, illuminance, correlated color temperature, PM2.5 concentration, CO2 concentration, and ambient noise.

[0037] Step 2: Preprocess the collected physiological and environmental data.

[0038] For physiological data, preprocessing includes filtering, denoising, artifact correction, outlier removal, windowing, and normalization. For environmental data, preprocessing includes time synchronization, missing value imputation, sliding time window aggregation, and normalization. Samples are formed using 20-second windows and 2-second steps. Physiological samples are extracted from the physiological data, and environmental variables within the same time window are averaged or statistically analyzed to obtain environmental samples that are time-aligned with the physiological samples.

[0039] Step 3: Perform unsupervised contrastive learning pre-training on the preprocessed physiological signals to obtain the pre-trained physiological encoder; First, construct two different enhanced views for the same sample. Let the physiological sample be... After data augmentation, two views are obtained. and ,Right now: in, and This represents a data augmentation function, where augmentation methods include, but are not limited to, one or more of the following: temporal slice rearrangement, random pruning, noise perturbation, amplitude scaling, and temporal masking. The augmented physiological samples are then input into a shared physiological coding network. Using a Long Short-Term Memory (LSTM) network, the corresponding physiological representations were obtained: Furthermore, physiological representations are input into the projection head. Using a multilayer perceptron (MLP), representations in the contrastive learning space are obtained: Within a batch, two augmented views of the same sample are treated as positive sample pairs, and the remaining samples are treated as negative sample pairs. A contrastive loss is used for training. For any positive sample pair... Its contrast loss can be expressed as: in, ·) is the similarity function, τ is the temperature coefficient, and N is the total number of samples in the batch.

[0040] Step 4: In the emotion recognition stage, the preprocessed physiological signals are input into the pre-trained physiological encoder to extract physiological representations; The physiological encoding network obtained by unsupervised contrastive learning pre-training Transferring to the classification stage, physiological samples enter Obtain physiological characterization .

[0041] Step 5: Input the preprocessed environmental sample into the thermal sensing characterization extraction module to extract the thermal sensing characterization; Let the environmental input sample be Extracting networks through thermal sensing representation Obtain thermal sensing characterization : in, This is an MLP (Multi-Level Processing) consisting of an input layer, at least one hidden layer, and an output layer. The hidden layer employs a fully connected structure and incorporates a non-linear activation function to map features to environmental variables.

[0042] Thermal sensing characterization This is an intermediate feature vector of environmental data in the latent space, used to characterize the subjective perception tendency of the target object to be hot or cold under the current environmental conditions. Then, a heat perception prediction head is added to predict heat perception labels, using cross-entropy loss.

[0043] Step 6: Fuse the physiological representation and thermal perception representation to obtain a joint representation.

[0044] Let physiological characteristics be Thermal sensing is characterized as Then the joint representation is denoted as : Using a gating fusion method, the gating vector is first calculated. : Then we obtain the joint characterization: in, Represents the activation function, preferably function; This represents element-wise multiplication; and These represent the learnable parameter matrix and the bias term, respectively.

[0045] Step 7: Input the joint representation into the emotion recognition model and output the emotion recognition result.

[0046] Let the emotion recognition model be Similarly, using an MLP classifier head, the emotion recognition result can be represented as: The classification loss uses cross-entropy loss. .

[0047] The overall training objective can be expressed as: in, For loss weighting coefficients, This is the loss function for thermal sensing prediction.

[0048] Thus, steps 1 through 7 complete an emotion recognition method that takes into account environmental perception.

[0049] The above is an introduction to the method embodiments. The following system embodiments will further illustrate the present invention.

[0050] This embodiment also provides an emotion recognition system that implements the above-described emotion recognition method that considers environmental perception, including: The data acquisition module is used to collect physiological data of the target object and environmental data of the environment in which the target object is located; The preprocessing module is used to preprocess the collected physiological and environmental data; The pre-training module is used to perform unsupervised contrastive learning pre-training on preprocessed physiological data to obtain a pre-trained physiological encoder. The physiological characterization extraction module is used to input preprocessed physiological data into a pre-trained physiological encoder to extract physiological characterization. The environmental perception representation extraction module is used to input preprocessed environmental data into the pre-built and trained environmental perception representation extraction module to extract environmental perception representations. The fusion module is used to fuse physiological representations and environmental perception representations to obtain a joint representation; The emotion recognition module is used to input the joint representation into a pre-built and trained emotion recognition module to obtain the emotion recognition result.

[0051] It should be noted that the specific content and beneficial effects of the system in this application can be found in the above method embodiments, and will not be repeated here.

[0052] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An emotion recognition method considering environmental perception, characterized in that, Includes the following steps: Collect physiological data of the target object and environmental data of the environment in which the target object is located; Preprocess the collected physiological and environmental data; Unsupervised contrastive learning pre-training is performed on the preprocessed physiological data to obtain a pre-trained physiological encoder; The preprocessed physiological data is input into the pre-trained physiological encoder to extract physiological representations; The preprocessed environmental data is input into the pre-built and trained environmental perception representation extraction module to extract the environmental perception representation; The physiological and environmental perception representations are fused to obtain a joint representation. The joint representation is input into a pre-built and trained emotion recognition module to obtain the emotion recognition result.

2. The emotion recognition method considering environmental perception according to claim 1, characterized in that, The physiological data includes one or more of the following: electrocardiogram (ECG) signals, acceleration signals, skin conductance signals, respiratory signals, and electroencephalogram (EEG) signals.

3. The emotion recognition method considering environmental perception according to claim 1, characterized in that, The environmental data includes at least temperature data and humidity data, and further includes one or more of illuminance, correlated color temperature, particulate matter concentration, carbon dioxide concentration, and noise.

4. The emotion recognition method considering environmental perception according to claim 1, characterized in that, The unsupervised contrastive learning pre-training of the preprocessed physiological data includes: Data augmentation is performed on the same preprocessed physiological data sample to construct at least two augmented views; Each enhanced view is input into a shared physiological coding network to obtain the corresponding physiological representation; Each physiological representation is input into the projection head to obtain feature representations in the contrastive learning space, and the consistency of the feature representations is constrained by contrastive loss, thereby pre-training the physiological coding network.

5. The emotion recognition method considering environmental perception according to claim 4, characterized in that, The contrast loss is an enhanced view derived from the same physiological data sample. and As positive sample pairs Enhanced view of the remaining physiological data samples As negative sample pairs This allows for the calculation of the contrast loss; For any positive sample pair The corresponding expression for calculating the contrast loss is: In the formula, Positive sample pairs The comparative loss, For similarity function, To enhance the view Feature representation, For temperature coefficient, N This represents the total number of samples within the batch.

6. The emotion recognition method considering environmental perception according to claim 4, characterized in that, The data augmentation methods include one or more of the following: time slice rearrangement, random pruning, noise perturbation, amplitude scaling, and time masking.

7. The emotion recognition method considering environmental perception according to claim 1, characterized in that, The environmental perception representation is either a thermal perception representation or a light perception representation.

8. The emotion recognition method considering environmental perception according to claim 7, characterized in that, The thermal perception representation is the intermediate feature vector of the environmental data in the latent space of the environmental perception representation extraction module, which is used to represent the target object's subjective perception tendency of hot and cold in the current environmental state. The environmental perception representation extraction module is a multilayer perceptron model; During the training process of the environmental perception representation extraction module, a thermal perception prediction head is added to predict the thermal perception state based on the thermal perception representation output by the environmental perception representation extraction module, and compare it with the corresponding label to calculate the loss function to optimize the parameters of the environmental perception representation extraction module.

9. The emotion recognition method considering environmental perception according to claim 1, characterized in that, The fusion method of the physiological representation and the environmental perception representation is any one of weighted fusion, gated fusion and attention fusion; The emotion recognition results include one or more of the following: arousal, valence, discrete emotion category, and continuous emotion value.

10. An emotion recognition system that implements an emotion recognition method considering environmental perception as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to collect physiological data of the target object and environmental data of the environment in which the target object is located; The preprocessing module is used to preprocess the collected physiological and environmental data; The pre-training module is used to perform unsupervised contrastive learning pre-training on preprocessed physiological data to obtain a pre-trained physiological encoder. The physiological characterization extraction module is used to input the pre-processed physiological data into the pre-trained physiological encoder to extract physiological characterization. The environmental perception representation extraction module is used to input preprocessed environmental data into the pre-built and trained environmental perception representation extraction module to extract environmental perception representations. The fusion module is used to fuse the physiological representation and the environmental perception representation to obtain a joint representation; An emotion recognition module is used to input the joint representation into a pre-built and trained emotion recognition module to obtain emotion recognition results.

Images