A system and method for inferring emotional state based on rppg signals

Abstract

The application relates to the field of artificial intelligence and emotion computing technology, and discloses a system and a method for inferring an emotional state based on an rPPG signal. The system comprises: a video acquisition module that acquires a facial video stream through an intelligent sensor; an rPPG signal extraction module that performs biological feature recognition and signal pattern recognition to extract an rPPG physiological signal; a group distribution reference module that stores a group baseline distribution model; an individual anchoring module that maps a resting rPPG physiological signal into a group standard space through pattern recognition to determine an anchoring coordinate; a disturbance mapping module that aligns the distribution according to the anchoring coordinate and combines standardized disturbance values into a multidimensional disturbance vector in a preset dimension order through pattern recognition; and an emotional state inference module that respectively outputs valence, arousal and dominance components through three branches of a three-dimensional emotion regression model to realize object recognition of a target emotional state.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, and in particular to an affective state inference system and method based on rPPG signals. Background Technology

[0002] With the rapid development of artificial intelligence and affective computing technologies, non-contact physiological signal emotion recognition based on facial videos has become a research hotspot in the field of human-computer interaction. Remote photoplethysmography (TP) technology analyzes micro-color changes in skin caused by heartbeats in facial videos, enabling non-contact extraction of pulse wave signals. From these signals, physiological features such as heart rate variability are extracted, and machine learning models are used to map these features into emotional state labels, providing technical support for applications such as mental health assessment, distance education, and smart cockpits.

[0003] However, existing methods for inferring emotional states based on rPPG signals face a fundamental technical bottleneck: the lack of a standardized mechanism for cross-individual emotional physiological expression. Due to physiological differences such as age, skin color, baseline heart rate, skin pigmentation, and vascular elasticity, individuals naturally exhibit different physiological response patterns in their rPPG signals when experiencing the same emotional state, both in absolute value and magnitude of variation. Existing methods only perform baseline calibration or simple normalization within the individual's own signal space. However, the rPPG signal characteristics exhibited by individual A under "high arousal" may be completely different numerically from those exhibited by individual B under "high arousal," preventing the emotional inference model from establishing a universal emotional physiological mapping relationship across individuals. This bottleneck significantly reduces the generalization ability of existing systems when facing new users, making reliable deployment in practical applications difficult. This invention reconstructs the entire problem as a standardized definition problem of emotional labels across physiological homeostasis. By referencing the group baseline distribution and mapping individual anchored coordinates, it fundamentally unifies the measurement standards for emotional physiological expression among different individuals.

[0004] Therefore, this invention proposes an emotion state inference system and method based on rPPG signals. Summary of the Invention

[0005] This invention provides an emotion state inference system and method based on rPPG signals. By establishing a dual-reference standardization mechanism of a group baseline distribution model and individual anchor coordinates, it fundamentally solves the cross-individual generalization barrier caused by the incomparability of emotion rPPG signal patterns among different individuals due to physiological differences. This enables emotion state inference to obtain a unified cross-individual measurement benchmark in the group standard space for the first time.

[0006] This invention provides an emotion state inference system based on rPPG signals, comprising: The video capture module is used to acquire facial video streams of the target individual through intelligent sensors; The rPPG signal extraction module is used to perform biometric recognition and signal pattern recognition from facial video streams and extract rPPG physiological signals. The population distribution reference module is used to store a population baseline distribution model constructed based on the rPPG physiological signals of multiple reference individuals under baseline conditions. The population baseline distribution model includes the population mean and population standard deviation for each rPPG physiological signal dimension. The individual anchoring module is used to acquire the resting rPPG physiological signal of the target individual in a resting state. Through pattern recognition, the resting rPPG physiological signal is mapped into the standard space defined by the population baseline distribution model to determine the anchoring coordinates of the target individual in the population distribution. The perturbation mapping module is used to perform distribution alignment processing on the rPPG physiological signals output in real time by the rPPG signal extraction module according to the anchor coordinates, generate standardized perturbation values ​​that are comparable across individuals in the standard space of the population baseline distribution model, and combine the standardized perturbation values ​​into a multidimensional perturbation vector according to a preset dimension order through pattern recognition. The emotion state inference module is used to input multidimensional perturbation vectors into a pre-trained three-dimensional emotion regression model. The three-dimensional emotion regression model is configured with valence regression branch, arousal regression branch and dominance regression branch. The three branches output valence component, arousal component and dominance component through pattern classification, respectively, and realize object recognition of target emotion state based on valence component, arousal component and dominance component.

[0007] Furthermore, the population distribution reference module includes: The group data acquisition unit is used to collect rPPG physiological signal samples from multiple reference individuals in the baseline state, where the reference individuals are sitting still with neutral facial expressions. The dimensional statistics unit is used to fit Gaussian distributions to rPPG physiological signal samples from multiple reference individuals in each dimension. The mean of the fit in each dimension is used as the population mean in the corresponding dimension, and the standard deviation of the fit in each dimension is used as the population standard deviation in the corresponding dimension. The distribution model storage unit is used to store the population mean and population standard deviation for each dimension as a population baseline distribution model by dimension index.

[0008] Furthermore, the individual anchoring module includes: The resting signal acquisition unit is used to acquire a fixed-duration resting facial video stream of the target individual in a resting state, and to extract the resting rPPG physiological signal from the resting facial video stream; The spatial mapping unit is used to take the mean value of each dimension in the resting rPPG physiological signal to obtain the resting mean value of the corresponding dimension. It calculates the difference between the resting mean value and the population mean value of the corresponding dimension in the population baseline distribution model, and divides the difference by the population standard deviation of the corresponding dimension to obtain the standardized coordinate components of the corresponding dimension. Finally, it combines the standardized coordinate components of all dimensions into anchor coordinates.

[0009] Furthermore, the perturbation mapping module includes: The distribution alignment unit is used to subtract the product obtained by multiplying the standardized coordinate component of the corresponding dimension in the anchor coordinates with the population standard deviation of the corresponding dimension in the population baseline distribution model from the value of each dimension in the rPPG physiological signal output in real time by the rPPG signal extraction module, so as to obtain the alignment value of each dimension. The standardized unit is used to divide the alignment value by the population standard deviation of the corresponding dimension in the population baseline distribution model to obtain the standardized perturbation value. The vector combination unit is used to arrange the standardized perturbation values ​​of all dimensions in a preset dimensional order to obtain a multidimensional perturbation vector.

[0010] Furthermore, the three-dimensional sentiment regression model includes an information isolation gating layer, which comprises: The dimension coupling matrix estimation unit is used to estimate the arousal-valence coupling strength factor, arousal-dominance coupling strength factor, and valence-dominance coupling strength factor of the current emotional state based on the conditional mutual information between the dimensions in the multidimensional perturbation vector of the current time frame and the number of consecutive time frames preceding the current time frame. The cross-inhibition unit is used to quantify the portion of the arousal-valence coupling strength factor that is higher than a preset coupling threshold into the input inhibition coefficient of the valence regression branch, and to quantify the portion of the arousal-dominance coupling strength factor that is higher than a preset coupling threshold into the input inhibition coefficient of the dominance regression branch. The input inhibition coefficients of the valence regression branch and the dominance regression branch are then applied to the feature input layer of their respective branches.

[0011] Furthermore, it also includes an artifact co-response analysis module, which includes: The artifact extraction unit is used to extract the inter-frame motion intensity signal of the facial region of interest during the processing of the facial video stream by the rPPG signal extraction module. The inter-frame motion intensity signal is a time series of pixel displacement amplitudes within the region of interest, calculated frame by frame. The co-response calculation unit is used to calculate the time-aligned sliding window cross-correlation function between the inter-frame motion intensity signal and the pulse wave amplitude envelope in the rPPG physiological signal when the system is in the sudden emotional event detection mode. Based on the time interval between the local peak of the sliding window cross-correlation function and the time when the falling edge of the pulse wave amplitude envelope appears, the autonomic nerve activation co-response reliability is obtained. The wake-up cross-validation unit is used to use the autonomic neural activation resonance confidence as a prior factor to perform confidence-weighted correction on the arousal component output by the emotion state inference module, and replace the original arousal component with the corrected arousal component as the arousal component used by the emotion state inference module to realize object recognition of the target emotion state.

[0012] Furthermore, it also includes an individual anchoring drift monitoring module, which includes: The drift assessment unit is used to periodically remap the latest resting rPPG physiological signal of the target individual into the standard space of the population baseline distribution model, and calculate the Euclidean distance between the new anchor coordinates and the original anchor coordinates as the anchor drift amount. The drift trigger unit is used to trigger an update operation of the anchor coordinates when the anchor drift exceeds a preset drift threshold.

[0013] Furthermore, it also includes a perturbation pattern library and a knowledge transfer module, which include: The pattern library storage unit is used to store high-quality multi-dimensional perturbation vectors and corresponding emotional state annotation results recorded in historical emotional events, forming a perturbation pattern library; The pattern retrieval unit is used to obtain the multi-dimensional perturbation vectors of the current time frame and several consecutive time frames before the current time frame from the emotional state inference module in the current emotional event as a query sequence. When the difference between the current anchor coordinates of the target individual and the anchor coordinates when the historical record was generated is less than a preset consistency threshold, the query sequence is directly matched with the historical records in the perturbation pattern library. When it is greater than or equal to the preset consistency threshold, the multi-dimensional perturbation vectors corresponding to the historical records in the perturbation pattern library are first inversely corrected to a space consistent with the current anchor coordinates based on the difference between the current anchor coordinates of the target individual and the anchor coordinates when the historical record was generated, and then the pattern matching is performed to retrieve the most similar preset number of historical records as similar perturbation cases. The prior fusion unit is used to take the emotional state labeling results corresponding to similar perturbation cases as prior information, and fuse them with the valence component, arousal component and dominance component output by the three-dimensional emotional regression model through Bayesian correction to generate fused valence component, fused arousal component and fused dominance component. The fused valence component, fused arousal component and fused dominance component replace the original output of the three-dimensional emotional regression model and are used by the emotional state inference module to realize object recognition of target emotional state.

[0014] Furthermore, it also includes a cross-individual generalization validation module, which includes: The clustering unit is used to perform hierarchical clustering on multiple target individuals based on the target individual anchoring coordinates generated by the individual anchoring module for each of the target individuals, and to obtain the individual clustering results. The generalization evaluation unit is used to calculate the intra-cluster cohesion and inter-cluster separation of the perturbation values ​​of each individual in each individual cluster within the standard space of the population baseline distribution model, and to evaluate the generalization performance of the three-dimensional sentiment regression model for sentiment inference of the corresponding individual clusters based on the intra-cluster cohesion and inter-cluster separation. The model update triggering unit is used to trigger the incremental retraining operation of the three-dimensional emotion regression model when the sentiment inference generalization performance of the individual cluster to which the target individual belongs is lower than the preset generalization performance threshold. This is done by using the rPPG physiological signals and sentiment state annotation results of multiple target individuals corresponding to the individual cluster to which the target individual belongs as supplementary training samples.

[0015] This invention provides a method for inferring emotional state based on rPPG signals, comprising the following steps: Biometric and signal pattern recognition are performed on facial video streams to extract rPPG physiological signals; Obtain a population baseline distribution model based on the rPPG physiological signals of multiple reference individuals under baseline conditions. The population baseline distribution model includes the population mean and population standard deviation for each rPPG physiological signal dimension. The resting rPPG physiological signal of the target individual was acquired in the resting state. The resting rPPG physiological signal was mapped into the standard space defined by the population baseline distribution model through pattern recognition to determine the anchor coordinates of the target individual in the population distribution. The real-time extracted rPPG physiological signals are processed for distribution alignment based on anchor coordinates to generate standardized perturbation values ​​that are comparable across individuals in the standard space of the population baseline distribution model. The standardized perturbation values ​​are then combined into a multidimensional perturbation vector according to a preset dimensional order through pattern recognition. The multidimensional perturbation vector is input into a pre-trained three-dimensional sentiment regression model, which is configured with valence regression branch, arousal regression branch and dominance regression branch; The valence component is output through the valence regression branch, the arousal component is output through the arousal regression branch, and the dominance component is output through the dominance regression branch. Object recognition of target emotional states is achieved based on valence, arousal, and dominance components.

[0016] The beneficial effects of this invention compared to existing technologies are as follows: This invention pre-constructs a population baseline distribution model as a unified cross-individual metric, and before each emotion inference, maps the resting rPPG physiological signal of the target individual into a standard space to obtain anchor coordinates. This allows the real-time rPPG physiological signal to be transformed into a standardized perturbation vector relative to the population baseline, thereby directly eliminating the incomparability of emotion signals caused by individual physiological differences. Based on this standardization mechanism, the perturbation vectors generated by the same emotional state of different individuals are comparable in the same coordinate space. The model does not need to readapt to the unknown signal distribution for new users and can directly output reliable emotion state quantification results, achieving a leap in cross-individual generalization performance. At the same time, this method of transforming emotion inference from individual dependence to a population-standardized metric provides a standardized input with consistency and stability within the population standard space for subsequent multidimensional emotion regression, significantly improving the accuracy and robustness of emotion state inference in real-world scenarios when facing users of different ages, skin colors, and baseline heart rates.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a system architecture diagram of the emotion state inference system based on rPPG signals in an embodiment of the present invention; Figure 2 This is a data processing flowchart in an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] refer to Figure 1 and Figure 2 This invention provides an embodiment of an emotion state inference system based on rPPG signals, comprising: The video capture module is used to acquire facial video streams of the target individual through intelligent sensors; The rPPG signal extraction module is used to perform biometric recognition and signal pattern recognition from facial video streams and extract rPPG physiological signals. The population distribution reference module is used to store a population baseline distribution model constructed based on the rPPG physiological signals of multiple reference individuals under baseline conditions. The population baseline distribution model includes the population mean and population standard deviation for each rPPG physiological signal dimension. The individual anchoring module is used to acquire the resting rPPG physiological signal of the target individual in a resting state. Through pattern recognition, the resting rPPG physiological signal is mapped into the standard space defined by the population baseline distribution model to determine the anchoring coordinates of the target individual in the population distribution. The perturbation mapping module is used to perform distribution alignment processing on the rPPG physiological signals output in real time by the rPPG signal extraction module according to the anchor coordinates, generate standardized perturbation values ​​that are comparable across individuals in the standard space of the population baseline distribution model, and combine the standardized perturbation values ​​into a multidimensional perturbation vector according to a preset dimension order through pattern recognition. The emotion state inference module is used to input multidimensional perturbation vectors into a pre-trained three-dimensional emotion regression model. The three-dimensional emotion regression model is configured with valence regression branch, arousal regression branch and dominance regression branch. The three branches output valence component, arousal component and dominance component through pattern classification, respectively, and realize object recognition of target emotion state based on valence component, arousal component and dominance component.

[0022] In this embodiment, acquiring the facial video stream of the target individual through a smart sensor means using an RGB camera module with integrated optical sensing elements as a video acquisition device. This module continuously captures images of the target individual's facial area at a fixed frame rate in the visible light band, and then arranges the captured image frames in chronological order to form a facial video stream. The camera module does not need to have physical contact with the target individual; the target individual only needs to face the camera naturally to complete the video acquisition.

[0023] In this embodiment, biometric recognition and signal pattern recognition are performed from the facial video stream to extract rPPG physiological signals. This involves first locating the facial region in each frame of the facial video stream using a face detection algorithm, and further identifying exposed skin areas such as the forehead and cheeks as regions of interest (ROIs). Then, the spatial average brightness values ​​of all pixels within the R, G, and B color channels of each RROI are statistically analyzed frame-by-frame to form a time-series signal across the three color channels. Finally, blind source separation is performed on the time-series signals of the three color channels to separate the blood volume change component synchronized with the pulse wave as the rPPG physiological signal. Biometric recognition in this process manifests as the localization and region extraction of the face, while signal pattern recognition involves identifying and separating the pulse wave pattern from the mixed color channel signal.

[0024] In this embodiment, the rPPG physiological signal includes four dimensions: pulse rate, pulse wave amplitude, pulse wave width, and heart rate variability frequency domain power. The pulse rate dimension is the number of pulses per minute detected from the rPPG physiological signal; the pulse wave amplitude dimension is the difference in amplitude between the peak and trough of the signal within each pulse cycle; the pulse wave width dimension is the time span between the rising edge and falling edge of the signal within each pulse cycle; and the heart rate variability frequency domain power dimension is the power integral value of the pulse interval time series after frequency domain transformation within a preset low-frequency band and a preset high-frequency band.

[0025] In this embodiment, the baseline state refers to the specific physiological and behavioral conditions maintained by the reference individual when the data is collected. Specifically, it refers to the state in which the reference individual sits quietly in front of the data collection device, maintains a neutral facial expression, does not make large head movements, does not engage in verbal communication, and does not come into contact with any audiovisual stimuli that may cause emotional fluctuations.

[0026] In this embodiment, the reference individual refers to a non-target individual volunteer who participates in constructing the population baseline distribution model. The selection of reference individuals needs to cover different age ranges, genders, skin color types, and baseline heart rate ranges to ensure the representativeness and breadth of coverage of the population baseline distribution model in terms of demographic dimensions.

[0027] In this embodiment, the resting state refers to the quiet and relaxed state maintained by the target individual before formal emotional state inference. Specifically, it refers to the state in which the target individual sits quietly in front of the data acquisition device, maintains a neutral facial expression, does not make large head movements, does not engage in verbal communication, and does not come into contact with any audiovisual stimuli that may cause emotional fluctuations. The resting state and the baseline state have the same behavioral requirements, the difference being that the resting state is for the target individual's data acquisition phase, while the baseline state is for the reference individual's data acquisition phase.

[0028] In this embodiment, acquiring the resting rPPG physiological signal of the target individual in a resting state refers to continuously acquiring a fixed-duration resting facial video stream using a smart sensor after the target individual enters a resting state. Then, the same biometric recognition and signal pattern recognition processing procedures as the rPPG signal extraction module are performed on the resting facial video stream to extract the target individual's resting rPPG physiological signal. In one possible implementation, the target individual actively triggers the acquisition of the resting facial video stream by clicking a confirmation button on the interactive interface, and the system uses the moment of user confirmation as the starting point for the acquisition of the resting state.

[0029] In this embodiment, pattern recognition maps resting rPPG physiological signals into a standard space defined by a population baseline distribution model to determine the anchor coordinates of the target individual within the population distribution. This involves first averaging the values ​​of each dimension of the target individual's resting rPPG physiological signal to obtain the resting mean for that dimension; then calculating the difference between the resting mean and the population mean for the corresponding dimension in the population baseline distribution model; and finally dividing the difference by the population standard deviation for that dimension to obtain the standardized coordinate components for that dimension; finally, combining the standardized coordinate components of all rPPG physiological signal dimensions into a multidimensional vector according to a preset dimensional order. This multidimensional vector represents the anchor coordinates of the target individual within the population distribution. Pattern recognition in this process involves comparing and matching the features of each dimension of the resting rPPG physiological signal with the population statistical features stored in the population baseline distribution model to identify the relative physiological position of the target individual within the population.

[0030] In this embodiment, the standard space defined by the population baseline distribution model refers to a mathematical space with the population mean of each rPPG physiological signal dimension as the origin and the population standard deviation of each rPPG physiological signal dimension as the scale unit. Within this standard space, a positive standardized coordinate component for any individual indicates that the individual's resting level in this dimension is higher than the population average; a negative standardized coordinate component indicates that the individual's resting level in this dimension is lower than the population average; the absolute value of the standardized coordinate component represents the degree of deviation from the population average.

[0031] In this embodiment, the rPPG physiological signals output in real time by the rPPG signal extraction module are aligned according to anchor coordinates to generate standardized perturbation values ​​that are comparable across individuals within the standard space of the population baseline distribution model. This involves first subtracting the product of the standardized coordinate component of the corresponding dimension in the anchor coordinates and the population standard deviation of that dimension in the population baseline distribution model from the value of each dimension of the real-time rPPG physiological signal, thus obtaining the alignment value for each dimension. Then, the alignment value for each dimension is divided by the population standard deviation of that dimension in the population baseline distribution model to obtain the standardized perturbation value for each dimension. After this processing, the standardized perturbation value represents the standardized deviation of the target individual's current rPPG physiological signal in the population standard space, after dual correction to both its own resting level and the population statistical distribution. The same emotional state in different target individuals will produce numerically comparable standardized perturbation values ​​within this standard space.

[0032] In this embodiment, constructing a multidimensional perturbation vector from standardized perturbation values ​​through pattern clustering refers to arranging the standardized perturbation values ​​of all rPPG physiological signal dimensions obtained after distribution alignment processing according to a preset dimensional order to form a multidimensional vector. This multidimensional perturbation vector is an ordered set of standardized perturbation values, where each element corresponds to the degree of perturbation in one rPPG physiological signal dimension. Pattern clustering in this process is manifested as the joint organization and structured dimensional clustering arrangement of perturbation features exhibited in different dimensions, forming a unified format of multidimensional feature representation.

[0033] In this embodiment, the pre-trained 3D emotion regression model refers to a multi-task regression deep learning model based on a long short-term memory network and fully connected layers. The model is constructed as follows: standardized perturbation vectors of multiple training individuals under emotionally induced states are collected as input training data, and manually labeled scores from a psychological valence-arousal-dominance 3D emotion scale are used as output training labels. Mean squared error is used as the loss function, and model parameters are optimized using backpropagation and adaptive momentum estimation algorithms. After training, the 3D emotion regression model is obtained. The shared underlying long short-term memory network has two or more hidden layers. Each of the three top branches contains a fully connected layer and a linear activation function output layer. The branch-specific fully connected layer receives the hidden state of the last time step output by the shared underlying long short-term memory network as input. The model's input is a multi-dimensional perturbation vector, and its output consists of three continuous real values: valence, arousal, and dominance components.

[0034] In this embodiment, the 3D sentiment regression model is configured with valence regression, arousal regression, and dominance regression branches. These three branches output valence, arousal, and dominance components respectively through pattern classification. This means that after extracting temporal features from a shared underlying long short-term memory network, the 3D sentiment regression model branches into three independent output branches at the top layer. The valence regression branch is the output head of the first fully connected layer with a linear activation function, specifically outputting the valence component; the arousal regression branch is the output head of the second fully connected layer with a linear activation function, specifically outputting the arousal component; and the dominance regression branch is the output head of the third fully connected layer with a linear activation function, specifically outputting the dominance component. Pattern classification in this process involves dividing the temporal perturbation features extracted from the bottom layer according to the semantic categories of the three sentiment dimensions, and then classifying the divided features into the corresponding dimensional output channels.

[0035] In this embodiment, the valence component, arousal component, and dominance component refer to the three-dimensional coordinate values ​​of the emotional state in the psychological valence-arousal-dominance three-dimensional emotional space. The valence component represents the positive or negative nature of the emotion, taking continuous real values ​​from negative to positive; the arousal component represents the activation intensity of the emotion, taking continuous real values ​​from low arousal to high arousal; and the dominance component represents the sense of control over the emotion, taking continuous real values ​​from controlled to autonomous. The combination of these three components forms the three-dimensional quantitative coordinates of the target emotional state.

[0036] In this embodiment, object recognition of the target emotional state based on valence, arousal, and dominance components refers to using the valence, arousal, and dominance components output by the three-dimensional emotion regression model as coordinates of the target emotional state in the valence-arousal-dominance three-dimensional emotion space. By looking up a pre-defined emotional semantic label mapping table, the spatial regions corresponding to the three-dimensional coordinates are mapped to specific emotional semantic labels, thus completing the object recognition of the target emotional state. The emotional semantic label mapping table predefines emotional semantic labels corresponding to different regions in the valence-arousal-dominance three-dimensional space. For example, a region with high valence, high arousal, and low dominance corresponds to the label "surprise," while a region with low valence, high arousal, and low dominance corresponds to the label "fear." The pre-defined emotional semantic label mapping table is constructed as follows: Valence, arousal, and dominance components of multiple training individuals under emotional induced states are collected, and discrete emotional semantic labels are labeled for each time frame by psychology experts. The three-dimensional space of valence-arousal-dominance is divided into positive and negative half-zones with zero as the boundary of valence dimension, high and low half-zones with the group mean as the boundary of arousal dimension, and high and low half-zones with the group mean as the boundary of dominance dimension, resulting in a total of eight spatial regions. The emotional semantic label that appears most frequently in each spatial region is used as the mapping label of that region.

[0037] Furthermore, the population distribution reference module includes: The group data acquisition unit is used to collect rPPG physiological signal samples from multiple reference individuals in the baseline state, where the reference individuals are sitting still with neutral facial expressions. The dimensional statistics unit is used to fit Gaussian distributions to rPPG physiological signal samples from multiple reference individuals in each dimension. The mean of the fit in each dimension is used as the population mean in the corresponding dimension, and the standard deviation of the fit in each dimension is used as the population standard deviation in the corresponding dimension. The distribution model storage unit is used to store the population mean and population standard deviation for each dimension as a population baseline distribution model by dimension index.

[0038] In this embodiment, collecting rPPG physiological signal samples from multiple reference individuals in their baseline state involves recruiting multiple volunteers who meet the selection criteria for reference individuals. Each reference individual is guided into the baseline state, and a fixed-duration baseline facial video stream is collected for each individual using a smart sensor. Biometric recognition and signal pattern recognition processes are then performed on each reference individual's baseline facial video stream to extract their rPPG physiological signal in the baseline state. Each reference individual's rPPG physiological signal is then considered as one rPPG physiological signal sample. The total number of reference individuals should ensure that each age range, gender, and skin color type cross-group contains a sufficient number of samples to guarantee the statistical representativeness of the subsequent Gaussian distribution fitting.

[0039] In this embodiment, a neutral facial expression means that the facial muscles are relaxed and do not exhibit facial features that can be identified as basic emotional categories such as happiness, sadness, anger, surprise, fear, and disgust. Specifically, it is characterized by naturally flat eyebrows, naturally open and close eyelids, naturally closed corners of the mouth, and no signs of tension or contraction in the facial muscles. This can be determined by the fact that the scores of all expression-related action units in the facial motion coding system are all below a preset threshold.

[0040] In this embodiment, Gaussian distribution fitting is performed on the rPPG physiological signal samples of multiple reference individuals dimension by dimension. This means that for the entire set of rPPG physiological signal samples from all reference individuals, all values ​​of each sample in the first dimension of the rPPG physiological signal are extracted sequentially, and a Gaussian distribution is fitted using the maximum likelihood estimation method. This process is repeated until Gaussian distribution fitting is completed for all dimensions of the rPPG physiological signal. During this process, it is not required that the Gaussian distribution fitted in each dimension must meet the preset shape parameters.

[0041] In this embodiment, the fitted mean for each dimension refers to the value corresponding to the center of symmetry of the Gaussian distribution curve for each dimension after fitting a Gaussian distribution to the rPPG physiological signal samples of multiple reference individuals dimension by dimension. This fitted mean serves as the population mean for that dimension in the population baseline distribution model.

[0042] In this embodiment, the fitting standard deviation for each dimension refers to the width parameter of the Gaussian distribution curve corresponding to each dimension after fitting a Gaussian distribution to the rPPG physiological signal samples of multiple reference individuals dimension by dimension. This fitting standard deviation serves as the population standard deviation for that dimension in the population baseline distribution model.

[0043] Furthermore, the individual anchoring module includes: The resting signal acquisition unit is used to acquire a fixed-duration resting facial video stream of the target individual in a resting state, and to extract the resting rPPG physiological signal from the resting facial video stream. The spatial mapping unit is used to take the mean value of each dimension in the resting rPPG physiological signal to obtain the resting mean value of the corresponding dimension. It calculates the difference between the resting mean value and the population mean value of the corresponding dimension in the population baseline distribution model, and divides the difference by the population standard deviation of the corresponding dimension to obtain the standardized coordinate components of the corresponding dimension. Finally, it combines the standardized coordinate components of all dimensions into anchor coordinates.

[0044] In this embodiment, the fixed duration refers to a pre-set continuous acquisition time, measured in seconds. The fixed duration must be set to ensure that a sufficient number of rPPG physiological signal cycles are fully acquired from the target or reference individual in the corresponding state, while avoiding changes in the state of the target or reference individual due to excessively long acquisition times. The fixed duration remains consistent throughout the population baseline distribution model construction phase and the individual anchoring phase.

[0045] In this embodiment, acquiring a fixed-duration resting facial video stream of the target individual in a resting state means that after the target individual enters a resting state, the facial area of ​​the target individual is continuously photographed by a smart sensor, and image frames of a fixed duration are continuously acquired. All the continuous image frames acquired within the fixed duration are then arranged in chronological order to form a resting facial video stream.

[0046] In this embodiment, extracting the resting rPPG physiological signal from the resting facial video stream involves performing face detection on the resting facial video stream to locate the facial region and identify the exposed skin area as the region of interest. The spatial average brightness values ​​of all pixels within the region of interest in the R, G, and B color channels are statistically analyzed frame-by-frame to form a time-series signal for each of the three color channels. Then, blind source separation processing is performed on the time-series signals of the three color channels to separate the blood volume change component synchronized with the pulse wave as the resting rPPG physiological signal. This processing flow is exactly the same as the processing flow of the rPPG signal extraction module for the facial video stream.

[0047] In this embodiment, the value of each dimension in the resting rPPG physiological signal refers to the value at each time point arranged in chronological order from the resting rPPG physiological signal. The dimensions of the resting rPPG physiological signal include pulse rate dimension, pulse wave amplitude dimension, pulse wave width dimension, and heart rate variability frequency domain power dimension. The value corresponding to each time sampling point in the time series of each dimension is the value of that dimension.

[0048] In this embodiment, the mean value of each dimension in the resting rPPG physiological signal is taken to obtain the resting mean value of the corresponding dimension. This means first determining the set of values ​​corresponding to all time sampling points of a dimension in the resting rPPG physiological signal within a fixed duration, then adding all the values ​​in the set and dividing by the total number of elements in the set. The resulting arithmetic mean is the resting mean value of that dimension. This operation is repeated for each dimension to obtain the resting mean values ​​of all dimensions.

[0049] In this embodiment, combining the standardized coordinate components of all dimensions into anchor coordinates means arranging the standardized coordinate components of all calculated rPPG physiological signal dimensions in a preset dimensional order that is the same as the dimensional index in the population baseline distribution model, forming a multidimensional vector. This multidimensional vector is the anchor coordinate of the target individual in the standard space defined by the population baseline distribution model.

[0050] Furthermore, the perturbation mapping module includes: The distribution alignment unit is used to subtract the product obtained by multiplying the standardized coordinate component of the corresponding dimension in the anchor coordinates with the population standard deviation of the corresponding dimension in the population baseline distribution model from the value of each dimension in the rPPG physiological signal output in real time by the rPPG signal extraction module, so as to obtain the alignment value of each dimension. The standardized unit is used to divide the alignment value by the population standard deviation of the corresponding dimension in the population baseline distribution model to obtain the standardized perturbation value. The vector combination unit is used to arrange the standardized perturbation values ​​of all dimensions in a preset dimensional order to obtain a multidimensional perturbation vector.

[0051] In this embodiment, the alignment value for each dimension is obtained by subtracting the product of the standardized coordinate component of the corresponding dimension in the anchor coordinates and the population standard deviation of the corresponding dimension in the population baseline distribution model from the value of each dimension in the real-time rPPG physiological signal output by the rPPG signal extraction module. This involves first determining the current value of one dimension in the real-time output rPPG physiological signal, then extracting the standardized coordinate component of that dimension from the anchor coordinates, multiplying the standardized coordinate component extracted from the anchor coordinates by the population standard deviation of that dimension in the population baseline distribution model to obtain the individual offset of that dimension, and then subtracting the individual offset from the current value of that dimension. This operation is repeated for each dimension to obtain the alignment value for all dimensions. The alignment value represents the signal value after translating the real-time rPPG physiological signal of the target individual from the individual physiological space to the population standard space.

[0052] In this embodiment, the preset dimension order refers to the unified order in which the various dimensions of the rPPG physiological signal are arranged during the construction phase of the population baseline distribution model. The preset dimension order is determined during the construction phase of the population baseline distribution model and is uniformly adopted during the generation of individual anchor coordinates, the construction of multidimensional perturbation vectors, and the input of the three-dimensional emotion regression model. All modules and units index dimensions according to this order to ensure the consistency of the dimension correspondence in the data flow throughout the entire system process.

[0053] In this embodiment, the standardized perturbation values ​​of all dimensions are arranged in a preset dimensional order to obtain a multidimensional perturbation vector. This means that the standardized perturbation values ​​of all rPPG physiological signal dimensions obtained after distribution alignment and standardization are arranged sequentially according to a preset dimensional order identical to the dimensional indices in the population baseline distribution model, forming a multidimensional vector, which is the multidimensional perturbation vector. The position of each element in the multidimensional perturbation vector corresponds one-to-one with the dimensional index in the population baseline distribution model. The three-dimensional sentiment regression model parses the meaning of each element in the input vector according to this dimensional order during both the training and inference phases.

[0054] Furthermore, the three-dimensional sentiment regression model includes an information isolation gating layer, which comprises: The dimension coupling matrix estimation unit is used to estimate the arousal-valence coupling strength factor, arousal-dominance coupling strength factor, and valence-dominance coupling strength factor of the current emotional state based on the conditional mutual information between the dimensions in the multidimensional perturbation vector of the current time frame and the number of consecutive time frames preceding the current time frame. The cross-inhibition unit is used to quantify the portion of the arousal-valence coupling strength factor that is higher than a preset coupling threshold into the input inhibition coefficient of the valence regression branch, and to quantify the portion of the arousal-dominance coupling strength factor that is higher than a preset coupling threshold into the input inhibition coefficient of the dominance regression branch. The input inhibition coefficients of the valence regression branch and the dominance regression branch are then applied to the feature input layer of their respective branches.

[0055] In this embodiment, the coupling strength factor of the current emotional state is estimated based on the conditional mutual information between the subsets of each dimension in the multidimensional perturbation vector. This means that the conditional mutual information values ​​between the arousal-related and valence-related subsets of the multidimensional perturbation vector are used as the arousal-valence coupling strength factor; the conditional mutual information values ​​between the arousal-related and dominance-related subsets are used as the arousal-dominance coupling strength factor; and the conditional mutual information values ​​between the valence-related and dominance-related subsets are used as the valence-dominance coupling strength factor. The arousal-related subset consists of the normalized perturbation values ​​of the pulse rate dimension and the heart rate variability low-frequency power dimension in the multidimensional perturbation vector; the valence-related subset consists of the normalized perturbation values ​​of the pulse wave amplitude dimension and the heart rate variability high-frequency power dimension; and the dominance-related subset consists of the normalized perturbation values ​​of the pulse wave width dimension and the heart rate variability frequency domain normalized dimension. The conditional mutual information is calculated based on the multidimensional perturbation vector sequence of the current time frame and its sixty preceding time frames.

[0056] In this embodiment, the preset coupling threshold refers to a pre-set critical value used to determine whether the currently estimated arousal-valence coupling strength factor or arousal-dominance coupling strength factor has become severe enough to require the initiation of cross-inhibition. The preset coupling threshold is determined by statistically analyzing the distribution of the arousal-valence coupling strength factor and the arousal-dominance coupling strength factor across all time frames in the training data during the training of the 3D emotion regression model, and taking the upper quartile of the distribution as the preset coupling threshold value.

[0057] In this embodiment, the input suppression coefficient of the valence regression branch is a multiplicative suppression factor ranging from zero to one, obtained by scaling the portion of the arousal-valence coupling strength factor that exceeds a preset coupling threshold. When the input suppression coefficient of the valence regression branch is zero, it indicates that there is no cross-interference from arousal to the valence branch that exceeds the allowable range, and the feature input layer of the valence regression branch remains unchanged. When the input suppression coefficient is not zero, each element of the feature input layer of the valence regression branch is multiplied by a coefficient less than one; the stronger the interference, the smaller the coefficient, thereby reducing the misleading effect of arousal signal crosstalk on valence prediction.

[0058] In this embodiment, the portion of the arousal-valence coupling strength factor exceeding a preset coupling threshold is quantified as the input inhibition coefficient of the valence regression branch. This involves subtracting the preset coupling threshold from the arousal-valence coupling strength factor; if the difference is positive, it is multiplied by a preset scaling factor to obtain the input inhibition coefficient of the valence regression branch; otherwise, the input inhibition coefficient of the valence regression branch is zero. The preset scaling factor is determined using a grid search method within the range of zero to one, based on the minimum mean square error criterion, during the training of the three-dimensional emotion regression model.

[0059] In this embodiment, the input suppression coefficient of the dominance regression branch is a multiplicative suppression factor ranging from zero to one, obtained by scaling the portion of the arousal-dominance coupling strength factor that exceeds a preset coupling threshold. When the input suppression coefficient of the dominance regression branch is zero, it indicates that there is no cross-interference from arousal to the dominance branch that exceeds the allowable range, and the feature input layer of the dominance regression branch remains unchanged. When the input suppression coefficient is not zero, each element of the feature input layer of the dominance regression branch is multiplied by a coefficient less than one; the stronger the interference, the smaller the coefficient, thereby reducing the misleading effect of arousal signal crosstalk on dominance prediction.

[0060] In this embodiment, the portion of the arousal-dominance coupling strength factor exceeding a preset coupling threshold is quantified as the input inhibition coefficient of the dominance regression branch. This involves subtracting the preset coupling threshold from the arousal-dominance coupling strength factor. If the difference is positive, it is multiplied by a preset scaling factor to obtain the input inhibition coefficient of the dominance regression branch. If the difference is not positive, the input inhibition coefficient of the dominance regression branch is set to zero. The preset scaling factor is determined using a grid search method within the range of zero to one, based on the minimum mean square error criterion, during the training of the 3D emotion regression model.

[0061] In this embodiment, applying the input suppression coefficients of the valence regression branch and the dominance regression branch to their respective feature input layers means multiplying the input suppression coefficient of the valence regression branch by the current value of each element in the feature input layer of the valence regression branch, and using the result to replace the original value as the updated value of the feature input layer of the valence regression branch; simultaneously, multiplying the input suppression coefficient of the dominance regression branch by the current value of each element in the feature input layer of the dominance regression branch, and using the result to replace the original value as the updated value of the feature input layer of the dominance regression branch. The feature input layer refers to the independent fully connected layer input vectors of the valence regression branch and the dominance regression branch. These input vectors are obtained by mapping the temporal features output from the shared underlying Long Short-Term Memory network through the branch-specific first fully connected layer.

[0062] Furthermore, it also includes an artifact co-response analysis module, which includes: The artifact extraction unit is used to extract the inter-frame motion intensity signal of the facial region of interest during the processing of the facial video stream by the rPPG signal extraction module. The inter-frame motion intensity signal is a time series of pixel displacement amplitudes within the region of interest, calculated frame by frame. The co-response calculation unit is used to calculate the time-aligned sliding window cross-correlation function between the inter-frame motion intensity signal and the pulse wave amplitude envelope in the rPPG physiological signal when the system is in the sudden emotional event detection mode. Based on the time interval between the local peak of the sliding window cross-correlation function and the time when the falling edge of the pulse wave amplitude envelope appears, the autonomic nerve activation co-response reliability is obtained. The wake-up cross-validation unit is used to use the autonomic neural activation resonance confidence as a prior factor to perform confidence-weighted correction on the arousal component output by the emotion state inference module, and replace the original arousal component with the corrected arousal component as the arousal component used by the emotion state inference module to realize object recognition of the target emotion state.

[0063] In this embodiment, the facial region of interest (ROI) refers to the exposed skin area automatically segmented in each frame of the facial video stream using a face detection algorithm and a facial landmark localization algorithm. Specifically, it includes the central forehead area and the left and right cheek areas. The selection criteria for the facial ROI are that the skin color consistency of the pixels within the area is higher than a preset consistency threshold, and the area does not contain non-skin feature areas such as eyebrows, eyes, nostrils, and lips.

[0064] In this embodiment, the time series of pixel displacement amplitude statistics in the region of interest by frame refers to the process of performing optical flow calculation on the region of interest of the face in two adjacent frames in the facial video stream to obtain the displacement vector of each pixel in the region of interest between the two adjacent frames. Then, the average value of the magnitude of the displacement vectors of all pixels in the region of interest is taken as the pixel displacement amplitude of the current frame. After repeating this operation frame by frame, the pixel displacement amplitudes of all frames are arranged in time order to form an inter-frame motion intensity signal.

[0065] In this embodiment, the pulse wave amplitude envelope in the rPPG physiological signal refers to the remaining signal after filtering out frequency components below 0.7 Hz and above 4 Hz from the time series of pulse wave amplitude dimension in the rPPG physiological signal through a bandpass filter, and then performing cubic spline interpolation on each local peak point of the filtered signal to obtain a continuous and smooth envelope curve, which is the pulse wave amplitude envelope.

[0066] In this embodiment, the system is in a sudden emotional event detection mode, which means that the emotional state inference module has detected that at least one emotional dimension has deviated from the baseline range, and the rate of this deviation exceeds a preset rate threshold. In this mode, the probability of autonomic co-response between facial motion signals and pulse wave amplitude changes is significantly higher than in a stable emotional state, and the artifact co-response analysis module is activated to provide cross-validation for the arousal component. The entry condition for the sudden emotional event detection mode is consistent with the initial judgment condition of the current emotional event, that is, at least one dimension of the valence component, arousal component, and dominance component output by the emotional state inference module deviates from the baseline range. The preset rate threshold is set to 0.5 units per second.

[0067] In this embodiment, calculating the time-aligned sliding window cross-correlation function between the inter-frame motion intensity signal and the pulse wave amplitude envelope in the rPPG physiological signal involves first setting a fixed-length sliding time window of five seconds. Each time, the sliding time window is synchronously moved backward by one sampling step (0.1 seconds) on both the inter-frame motion intensity signal and the pulse wave amplitude envelope. At each rest point within the sliding time window, the cross-correlation value between the inter-frame motion intensity signal and the pulse wave amplitude envelope within a preset time delay range (zero to 0.5 seconds) is calculated. The cross-correlation function is calculated using the Pearson correlation coefficient formula.

[0068] In this embodiment, the local peak value of the cross-correlation function of the sliding window refers to the maximum value of the cross-correlation values ​​among all the cross-correlation values ​​corresponding to the time delays calculated at each dwell position of the sliding time window, and the maximum value is greater than the preset lower limit threshold of 0.3.

[0069] In this embodiment, the time when the falling edge of the pulse wave amplitude envelope appears refers to the time when the amplitude value decreases by 50% of the total decrease in the falling segment between the local peak point of the waveform and the next adjacent local valley point on the pulse wave amplitude envelope curve.

[0070] In this embodiment, the time interval between the local peak of the cross-correlation function of the sliding window and the appearance of the falling edge of the pulse wave amplitude envelope is the time interval obtained by subtracting the time delay value corresponding to the local peak of the cross-correlation function of the sliding window at each dwell position of the sliding time window from the time difference between the appearance of the falling edge of the waveform closest to the local peak in the pulse wave amplitude envelope within the same sliding time window and the start time of the sliding window, and taking the absolute value of the difference.

[0071] In this embodiment, the autonomic activation resonance reliability is obtained based on the time interval between the local peak of the sliding window cross-correlation function and the appearance of the falling edge of the pulse wave amplitude envelope. This involves first dividing the value of the local peak of the sliding time window by a preset upper limit benchmark value of one to obtain a correlation score ranging from zero to one; then comparing the time interval between the local peak of the sliding window cross-correlation function and the appearance of the falling edge of the pulse wave amplitude envelope with a preset time alignment tolerance of 0.1 seconds. When the time interval does not exceed 0.1 seconds, the time alignment score is one; when the time interval exceeds 0.1 seconds but does not exceed 0.2 seconds, the time alignment score is 0.5; and when the time interval exceeds 0.2 seconds, the time alignment score is zero. Finally, the product of the correlation score and the time alignment score is used as the autonomic activation resonance reliability at the position of the sliding time window.

[0072] In this embodiment, the autonomic activation resonance reliability is used as a priori factor to perform a reliability-weighted correction on the arousal component output by the emotional state inference module. This means multiplying the original arousal component output by the arousal regression branch in the emotional state inference module by a coefficient, and adding the product of the autonomic activation resonance reliability and the portion of the original arousal component that exceeds the arousal benchmark value. The arousal benchmark value is the arousal component output value of the target individual in a resting state. If the target individual has not yet completed the resting state data collection, the arousal benchmark value is zero. The corrected arousal component is positively enhanced when the autonomic activation resonance reliability is high and the original arousal deviates significantly from the benchmark value, while the original output is retained when the autonomic activation resonance reliability is low.

[0073] Furthermore, it also includes an individual anchoring drift monitoring module, which includes: The drift assessment unit is used to periodically remap the latest resting rPPG physiological signal of the target individual into the standard space of the population baseline distribution model, and calculate the Euclidean distance between the new anchor coordinates and the original anchor coordinates as the anchor drift amount. The drift trigger unit is used to trigger an update operation of the anchor coordinates when the anchor drift exceeds a preset drift threshold.

[0074] In this embodiment, the latest resting rPPG physiological signal of the target individual is periodically remapped into the standard space of the population baseline distribution model. The Euclidean distance between the new anchor coordinates and the original anchor coordinates is calculated as the anchor drift. This means that, according to a preset monitoring cycle, at the time node of each monitoring cycle, the target individual is guided back into a resting state and a fixed duration of resting facial video stream is collected. The latest resting rPPG physiological signal of the target individual is extracted. Then, the value of each dimension of the latest resting rPPG physiological signal is averaged to obtain the latest resting mean. The difference between the latest resting mean and the population mean of that dimension in the population baseline distribution model is calculated. The difference is then divided by the population standard deviation of that dimension to obtain the latest standardized coordinate component of that dimension. The latest standardized coordinate components of all dimensions are combined to form the latest anchor coordinates. Finally, the sum of squares of the differences between the latest anchor coordinates and the original anchor coordinates stored in the system in each corresponding dimension is calculated, and the square root of the sum of squares is taken to obtain the anchor drift. The preset monitoring cycle is seven days.

[0075] In this embodiment, the preset drift threshold refers to a pre-set critical value used to determine whether the anchor coordinates of the target individual have undergone a significant change that requires a system response. The preset drift threshold is determined by statistically analyzing the anchor drift distribution of multiple reference individuals in the same monitoring period within a population baseline distribution model, and taking 1.5 times the upper quartile of this distribution as the preset drift threshold value.

[0076] In this embodiment, triggering the anchor coordinate update operation means that when the calculated anchor drift exceeds the preset drift threshold, the system automatically replaces the original anchor coordinates stored in the system with the latest calculated anchor coordinates of the target individual. The replaced anchor coordinates are used as the anchor coordinates for distribution alignment processing by the subsequent perturbation mapping module and for emotion inference by the emotion state inference module.

[0077] Furthermore, it also includes a perturbation pattern library and a knowledge transfer module, which include: The pattern library storage unit is used to store high-quality multi-dimensional perturbation vectors and corresponding emotional state annotation results recorded in historical emotional events, forming a perturbation pattern library; The pattern retrieval unit is used to obtain the multi-dimensional perturbation vectors of the current time frame and several consecutive time frames before the current time frame from the emotional state inference module in the current emotional event as a query sequence. When the difference between the current anchor coordinates of the target individual and the anchor coordinates when the historical record was generated is less than a preset consistency threshold, the query sequence is directly matched with the historical records in the perturbation pattern library. When it is greater than or equal to the preset consistency threshold, the multi-dimensional perturbation vectors corresponding to the historical records in the perturbation pattern library are first inversely corrected to a space consistent with the current anchor coordinates based on the difference between the current anchor coordinates of the target individual and the anchor coordinates when the historical record was generated, and then the pattern matching is performed to retrieve the most similar preset number of historical records as similar perturbation cases. The prior fusion unit is used to take the emotional state labeling results corresponding to similar perturbation cases as prior information, and fuse them with the valence component, arousal component and dominance component output by the three-dimensional emotional regression model through Bayesian correction to generate fused valence component, fused arousal component and fused dominance component. The fused valence component, fused arousal component and fused dominance component replace the original output of the three-dimensional emotional regression model and are used by the emotional state inference module to realize object recognition of target emotional state.

[0078] In this embodiment, a historical emotional event refers to a complete period of time, from the onset of a change in emotional state to its return to the baseline level, that has been fully recorded and confirmed by the system during the target individual's past use of this system for emotional state inference. The start time of a historical emotional event is the point at which at least one of the valence, arousal, and dominance components output by the emotional state inference module deviates from the baseline range, and the end time is the point at which all three dimensions return to the baseline range.

[0079] In this embodiment, the high-quality multidimensional perturbation vectors and corresponding emotional state annotation results recorded in historical emotional events refer to the multidimensional perturbation vectors of time frames within the time range covered by a confirmed historical emotional event. These vectors are labeled as high-quality multidimensional perturbation vectors, specifically those corresponding to time frames where the autonomic neural activation resonance confidence level is higher than a preset quality standard threshold. These high-quality multidimensional perturbation vectors, along with the valence, arousal, and dominance components confirmed by the target individual through an interactive interface or manually annotated by experts in the historical emotional event, are stored together as the corresponding emotional state annotation results. The preset quality standard threshold is 0.6.

[0080] In this embodiment, the current emotional event refers to the ongoing and unfinished emotional state change process that the target individual is currently experiencing. The method for determining the start of the current emotional event is consistent with that for historical emotional events, i.e., the time point when at least one dimension of the valence component, arousal component, and dominance component output by the emotional state inference module deviates from the baseline range is taken as the start time of the current emotional event.

[0081] In this embodiment, the multiple consecutive time frames preceding the current time frame refers to a set of time frames that trace back a preset number of consecutive time frames from the current time frame as the cutoff point. The preset number of frames is twenty frames. Each time frame in this set of time frames corresponds to a multidimensional perturbation vector generated by the perturbation mapping module at that time frame.

[0082] In this embodiment, the historical records in the perturbation pattern library refer to a complete record consisting of all high-quality multidimensional perturbation vectors corresponding to each historical emotional event stored in the perturbation pattern library and their corresponding emotional state annotation results. Each historical record contains a set of high-quality multidimensional perturbation vector sequences arranged in chronological order and the corresponding valence, arousal, and dominance component annotation values ​​of the sequence.

[0083] In this embodiment, pattern matching is performed between the query sequence and historical records in the perturbation pattern library to retrieve the most similar historical records as similar perturbation cases. This means: first, it is determined whether the difference between the current anchor coordinates and the target anchor coordinates of the target individual is less than a preset consistency threshold. The target anchor coordinates are the anchor coordinates of the target individual when the historical record was generated. If the difference is less than the preset consistency threshold, the multidimensional perturbation vectors of each time frame in the query sequence are directly concatenated end-to-end in chronological order to form a one-dimensional query feature vector. The high-quality multidimensional perturbation vector sequences of each historical record in the perturbation pattern library are concatenated in the same way to form a one-dimensional historical feature vector. The dynamic time warping distance is calculated, and the first preset number of historical records after sorting are taken as similar perturbation cases. If the difference is greater than or equal to the preset consistency threshold, the coordinates of the query sequence or historical records are first reverse-corrected based on the difference between the current anchor coordinates and the target anchor coordinates, and then the end-to-end concatenation and dynamic time warping distance calculation are performed. The preset consistency threshold is set to 0.1. When the anchor coordinates of a target individual are updated due to drift, the historical records generated under the old anchor coordinates will no longer participate in the pattern retrieval and matching under the current anchor coordinates, until the perturbation vector corresponding to the historical record is recalculated and updated into the database based on the new anchor coordinates.

[0084] In this embodiment, the preset quantity refers to a pre-defined integer value used to specify the maximum number of similar disturbance cases retrieved from the disturbance pattern library. The preset quantity is five.

[0085] In this embodiment, the emotional state labeling results corresponding to similar perturbation cases are used as prior information. These are then fused with the valence, arousal, and dominance components output by the 3D emotional regression model through Bayesian correction to generate fused valence, arousal, and dominance components. Specifically, the arithmetic mean of the valence component labels corresponding to a predetermined number of similar perturbation cases is taken as the prior valence mean, and the standard deviation of the valence component labels is taken as the prior valence standard deviation. This constructs a prior Gaussian distribution centered on the prior valence mean and with the prior valence standard deviation as its spread width. Next, the valence component output by the 3D emotional regression model is used as the mean of the likelihood Gaussian distribution, and the standard deviation of the valence prediction error of the 3D emotional regression model on the training set is used as the standard deviation of the likelihood Gaussian distribution. Finally, the prior Gaussian distribution and the likelihood Gaussian distribution are multiplied and fused according to Bayes' theorem, and the mean of the fused distribution is taken as the fused valence component. The arousal component and the dominance component were obtained by the same method to obtain the fused arousal component and the fused dominance component.

[0086] In this embodiment, the fused valence component, fused arousal component, and fused dominance component replace the original output of the three-dimensional emotion regression model and are used by the emotion state inference module to realize object recognition of the target emotion state. This means that after completing the Bayesian correction and generating the fused valence component, fused arousal component, and fused dominance component, the system no longer uses the original output value of the three-dimensional emotion regression model. Instead, it uses the three fused components as the final output of the emotion state inference module and passes them to the object recognition stage. The object recognition of the target emotion state is completed by looking up the preset emotion semantic label mapping table.

[0087] Furthermore, it also includes a cross-individual generalization validation module, which includes: The clustering unit is used to perform hierarchical clustering on multiple target individuals based on the target individual anchoring coordinates generated by the individual anchoring module for each of the target individuals, and to obtain the individual clustering results. The generalization evaluation unit is used to calculate the intra-cluster cohesion and inter-cluster separation of the perturbation values ​​of each individual in each individual cluster within the standard space of the population baseline distribution model, and to evaluate the generalization performance of the three-dimensional sentiment regression model for sentiment inference of the corresponding individual clusters based on the intra-cluster cohesion and inter-cluster separation. The model update triggering unit is used to trigger the incremental retraining operation of the three-dimensional emotion regression model when the sentiment inference generalization performance of the individual cluster to which the target individual belongs is lower than the preset generalization performance threshold. This is done by using the rPPG physiological signals and sentiment state annotation results of multiple target individuals corresponding to the individual cluster to which the target individual belongs as supplementary training samples.

[0088] In this embodiment, based on the target individual anchor coordinates generated by the individual anchoring module for each of the multiple target individuals, hierarchical clustering is performed on the multiple target individuals to obtain the individual cluster partitioning results. This involves first collecting the anchor coordinates corresponding to all registered target individuals in the system, with each anchor coordinate being a multi-dimensional vector. Then, each anchor coordinate is considered as an independent initial cluster, and the Euclidean distance between any two clusters is calculated. The two closest clusters are merged into a new cluster, and the coordinates of the new cluster are taken as the average of the coordinates of the two merged clusters. This merging process is repeated until the total number of clusters is reduced to a preset number. The final cluster is an individual cluster, and the target individuals contained within the cluster are the members of that individual cluster. The preset number of clusters is the integer part of the square root of the total number of multiple target individuals.

[0089] In this embodiment, calculating the intra-cluster cohesion and inter-cluster separation of the perturbation values ​​of each individual within each cluster in the standard space of the population baseline distribution model refers to, after obtaining the individual cluster division results, collecting the multidimensional perturbation vectors of each target individual in the cluster across all time frames of the most recent complete emotional event, calculating the mean of the Euclidean distances between all pairs of multidimensional perturbation vectors within the cluster, and using this mean as the intra-cluster compactness. A lower intra-cluster compactness indicates a more consistent perturbation response pattern among individuals within the cluster. Calculating the Euclidean distance between the cluster center vector of the cluster and the cluster center vectors of every other individual cluster, and taking the mean of all Euclidean distances as the inter-cluster separation, indicates a more significant distinction between the perturbation response pattern of the cluster and other individual clusters. The cluster center vector is a multidimensional vector composed of the arithmetic mean of all multidimensional perturbation vectors within the cluster.

[0090] In this embodiment, evaluating the generalization performance of the three-dimensional sentiment regression model for the corresponding individual cluster based on intra-cluster cohesion and inter-cluster separation means dividing the difference between the inter-cluster separation and intra-cluster compactness of the corresponding individual cluster by the inter-cluster separation, obtaining a ratio that takes a value in the range of zero to one. This ratio is used as a quantitative value of the generalization performance of the three-dimensional sentiment regression model for the individual cluster. The closer the ratio is to one, the higher the generalization performance of the model for the individual cluster.

[0091] In this embodiment, the preset generalization performance threshold refers to a pre-set critical value used to determine whether the generalization performance of the three-dimensional sentiment regression model for a certain individual cluster has reached a level that requires triggering a model update. The preset generalization performance threshold is set as 80% of the lowest value of the quantified generalization performance among all individual clusters after the system completes clustering and generalization performance evaluation of the first batch of registered target individuals after initial deployment.

[0092] In this embodiment, the rPPG physiological signals and emotional state annotations of multiple target individuals corresponding to the target individual's cluster are used as supplementary training samples to trigger incremental retraining of the 3D emotion regression model. This means that when the generalization performance quantization value of the emotion inference of the target individual's cluster is lower than a preset generalization performance threshold, the multidimensional perturbation vectors from the historical emotional events of all target individuals in that cluster are used as supplementary training input samples, and the corresponding emotional state annotations are used as supplementary training label samples. Based on the existing parameters of the 3D emotion regression model, the supplementary training samples are mixed with the original training samples at a preset mixing ratio. An adaptive momentum estimation algorithm is used to further optimize the model parameters. After optimization, the updated model parameters replace the original parameters of the 3D emotion regression model in the emotion state inference module. The preset mixing ratio is 30% of the total number of samples in each training batch.

[0093] This invention provides an embodiment of an emotion state inference method based on rPPG signals, comprising the following steps: Biometric and signal pattern recognition are performed on facial video streams to extract rPPG physiological signals; Obtain a population baseline distribution model based on the rPPG physiological signals of multiple reference individuals under baseline conditions. The population baseline distribution model includes the population mean and population standard deviation for each rPPG physiological signal dimension. The resting rPPG physiological signal of the target individual was acquired in the resting state. The resting rPPG physiological signal was mapped into the standard space defined by the population baseline distribution model through pattern recognition to determine the anchor coordinates of the target individual in the population distribution. The real-time extracted rPPG physiological signals are processed for distribution alignment based on anchor coordinates to generate standardized perturbation values ​​that are comparable across individuals in the standard space of the population baseline distribution model. The standardized perturbation values ​​are then combined into a multidimensional perturbation vector according to a preset dimensional order through pattern recognition. The multidimensional perturbation vector is input into a pre-trained three-dimensional sentiment regression model, which is configured with valence regression branch, arousal regression branch and dominance regression branch; The valence component is output through the valence regression branch, the arousal component is output through the arousal regression branch, and the dominance component is output through the dominance regression branch. Object recognition of target emotional states is achieved based on valence, arousal, and dominance components.

[0094] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. An emotion state inference system based on rPPG signals, characterized in that, include: The video capture module is used to acquire facial video streams of the target individual through intelligent sensors; The rPPG signal extraction module is used to perform biometric recognition and signal pattern recognition from facial video streams and extract rPPG physiological signals. The population distribution reference module is used to store a population baseline distribution model constructed based on the rPPG physiological signals of multiple reference individuals under baseline conditions. The population baseline distribution model includes the population mean and population standard deviation for each rPPG physiological signal dimension. The individual anchoring module is used to acquire a fixed-duration resting facial video stream of the target individual in a resting state, and extract the resting rPPG physiological signal from the resting facial video stream; the mean value of each dimension in the resting rPPG physiological signal is taken to obtain the resting mean value of the corresponding dimension, the difference between the resting mean value and the population mean value of the corresponding dimension in the population baseline distribution model is calculated, and the difference is divided by the population standard deviation of the corresponding dimension to obtain the standardized coordinate component of the corresponding dimension. Finally, the standardized coordinate components of all dimensions are combined into the anchoring coordinates. The perturbation mapping module is used to perform distribution alignment processing on the rPPG physiological signals output in real time by the rPPG signal extraction module according to the anchor coordinates, generate standardized perturbation values ​​that are comparable across individuals in the standard space of the population baseline distribution model, and arrange the standardized perturbation values ​​of all rPPG physiological signal dimensions obtained after distribution alignment processing in a preset dimensional order to form a multidimensional vector as a multidimensional perturbation vector. The emotion state inference module is used to input multidimensional perturbation vectors into a pre-trained three-dimensional emotion regression model. The three-dimensional emotion regression model is configured with valence regression branch, arousal regression branch and dominance regression branch. The three branches output valence component, arousal component and dominance component through pattern classification, respectively, and realize object recognition of target emotion state based on valence component, arousal component and dominance component.

2. The emotion state inference system based on rPPG signals as described in claim 1, characterized in that, The population distribution reference module includes: The group data acquisition unit is used to collect rPPG physiological signal samples from multiple reference individuals in the baseline state, where the reference individuals are sitting still with neutral facial expressions. The dimensional statistics unit is used to fit Gaussian distributions to rPPG physiological signal samples from multiple reference individuals in each dimension. The mean of the fit in each dimension is used as the population mean in the corresponding dimension, and the standard deviation of the fit in each dimension is used as the population standard deviation in the corresponding dimension. The distribution model storage unit is used to store the population mean and population standard deviation for each dimension as a population baseline distribution model by dimension index.

3. The emotion state inference system based on rPPG signals as described in claim 1, characterized in that, The disturbance mapping module includes: The distribution alignment unit is used to subtract the product obtained by multiplying the standardized coordinate component of the corresponding dimension in the anchor coordinates with the population standard deviation of the corresponding dimension in the population baseline distribution model from the value of each dimension in the rPPG physiological signal output in real time by the rPPG signal extraction module, so as to obtain the alignment value of each dimension. The standardized unit is used to divide the alignment value by the population standard deviation of the corresponding dimension in the population baseline distribution model to obtain the standardized perturbation value. The vector combination unit is used to arrange the standardized perturbation values ​​of all dimensions in a preset dimensional order to obtain a multidimensional perturbation vector.

4. The emotion state inference system based on rPPG signals as described in claim 1, characterized in that, The 3D sentiment regression model includes an information isolation gating layer, which comprises: The dimension coupling matrix estimation unit is used to estimate the arousal-valence coupling strength factor, arousal-dominance coupling strength factor, and valence-dominance coupling strength factor of the current emotional state based on the conditional mutual information between the dimensions in the multidimensional perturbation vector of the current time frame and the number of consecutive time frames preceding the current time frame. The cross-inhibition unit is used to quantify the portion of the arousal-valence coupling strength factor that is higher than a preset coupling threshold into the input inhibition coefficient of the valence regression branch, and to quantify the portion of the arousal-dominance coupling strength factor that is higher than a preset coupling threshold into the input inhibition coefficient of the dominance regression branch. The input inhibition coefficients of the valence regression branch and the dominance regression branch are then applied to the feature input layer of their respective branches.

5. The emotion state inference system based on rPPG signals as described in claim 1, characterized in that, It also includes an artifact co-response analysis module, which includes: The artifact extraction unit is used to extract the inter-frame motion intensity signal of the facial region of interest during the processing of the facial video stream by the rPPG signal extraction module. The inter-frame motion intensity signal is a time series of pixel displacement amplitudes within the region of interest, calculated frame by frame. The co-response calculation unit is used to calculate the time-aligned sliding window cross-correlation function between the inter-frame motion intensity signal and the pulse wave amplitude envelope in the rPPG physiological signal when the system is in the sudden emotional event detection mode. Based on the time interval between the local peak of the sliding window cross-correlation function and the time when the falling edge of the pulse wave amplitude envelope appears, the autonomic nerve activation co-response reliability is obtained. The wake-up cross-validation unit is used to use the autonomic neural activation resonance confidence as a prior factor to perform confidence-weighted correction on the arousal component output by the emotion state inference module, and replace the original arousal component with the corrected arousal component as the arousal component used by the emotion state inference module to realize object recognition of the target emotion state.

6. The emotion state inference system based on rPPG signals as described in claim 1, characterized in that, It also includes an individual anchoring drift monitoring module, which includes: The drift assessment unit is used to periodically remap the latest resting rPPG physiological signal of the target individual into the standard space of the population baseline distribution model, and calculate the Euclidean distance between the new anchor coordinates and the original anchor coordinates as the anchor drift amount. The drift trigger unit is used to trigger an update operation of the anchor coordinates when the anchor drift exceeds a preset drift threshold.

7. The emotion state inference system based on rPPG signals as described in claim 1, characterized in that, It also includes a perturbation pattern library and a knowledge transfer module, which include: The pattern library storage unit is used to store high-quality multi-dimensional perturbation vectors and corresponding emotional state annotation results recorded in historical emotional events, forming a perturbation pattern library; The pattern retrieval unit is used to obtain the multi-dimensional perturbation vectors of the current time frame and several consecutive time frames before the current time frame from the emotional state inference module in the current emotional event as a query sequence. When the difference between the current anchor coordinates of the target individual and the anchor coordinates when the historical record was generated is less than a preset consistency threshold, the query sequence is directly matched with the historical records in the perturbation pattern library. When it is greater than or equal to the preset consistency threshold, the multi-dimensional perturbation vectors corresponding to the historical records in the perturbation pattern library are first inversely corrected to a space consistent with the current anchor coordinates based on the difference between the current anchor coordinates of the target individual and the anchor coordinates when the historical record was generated, and then the pattern matching is performed to retrieve the most similar preset number of historical records as similar perturbation cases. The prior fusion unit is used to take the emotional state labeling results corresponding to similar perturbation cases as prior information, and fuse them with the valence component, arousal component and dominance component output by the three-dimensional emotional regression model through Bayesian correction to generate fused valence component, fused arousal component and fused dominance component. The fused valence component, fused arousal component and fused dominance component replace the original output of the three-dimensional emotional regression model and are used by the emotional state inference module to realize object recognition of target emotional state.

8. The emotion state inference system based on rPPG signals as described in claim 1, characterized in that, It also includes a cross-individual generalization validation module, which includes: The clustering unit is used to perform hierarchical clustering on multiple target individuals based on the target individual anchoring coordinates generated by the individual anchoring module for each of the target individuals, and to obtain the individual clustering results. The generalization evaluation unit is used to calculate the intra-cluster cohesion and inter-cluster separation of the perturbation values ​​of each individual in each individual cluster within the standard space of the population baseline distribution model, and to evaluate the generalization performance of the three-dimensional sentiment regression model for sentiment inference of the corresponding individual clusters based on the intra-cluster cohesion and inter-cluster separation. The model update triggering unit is used to trigger the incremental retraining operation of the three-dimensional emotion regression model when the sentiment inference generalization performance of the individual cluster to which the target individual belongs is lower than the preset generalization performance threshold. This is done by using the rPPG physiological signals and sentiment state annotation results of multiple target individuals corresponding to the individual cluster to which the target individual belongs as supplementary training samples.

9. A method for inferring emotional state based on rPPG signals, characterized in that, Includes the following steps: Biometric and signal pattern recognition are performed on facial video streams to extract rPPG physiological signals; Obtain a population baseline distribution model based on the rPPG physiological signals of multiple reference individuals under baseline conditions. The population baseline distribution model includes the population mean and population standard deviation for each rPPG physiological signal dimension. A fixed-duration resting facial video stream of the target individual in a resting state is acquired, and resting rPPG physiological signals are extracted from the resting facial video stream; the mean value of each dimension in the resting rPPG physiological signal is taken to obtain the resting mean value of the corresponding dimension; the difference between the resting mean value and the population mean value of the corresponding dimension in the population baseline distribution model is calculated, and the difference is divided by the population standard deviation of the corresponding dimension to obtain the standardized coordinate components of the corresponding dimension; finally, the standardized coordinate components of all dimensions are combined into anchor coordinates. The real-time extracted rPPG physiological signals are distributed and aligned according to the anchor coordinates to generate standardized perturbation values ​​that are comparable across individuals in the standard space of the population baseline distribution model. The standardized perturbation values ​​of all rPPG physiological signal dimensions obtained after distribution alignment are arranged in a preset dimensional order to form a multidimensional vector as a multidimensional perturbation vector. The multidimensional perturbation vector is input into a pre-trained three-dimensional sentiment regression model, which is configured with valence regression branch, arousal regression branch and dominance regression branch; The valence component is output through the valence regression branch, the arousal component is output through the arousal regression branch, and the dominance component is output through the dominance regression branch. Object recognition of target emotional states is achieved based on valence, arousal, and dominance components.

Images