An emotion perception method and system based on millimeter wave radar

Abstract

The application relates to the technical field of emotion recognition and human-computer interaction, and discloses an emotion perception method and system based on a millimeter wave radar, the method comprising the following steps: obtaining radar original data and sleep state context and performing demodulation baseband processing to obtain basic timing data; performing sliding window mean calculation and timing feature extraction on the basic timing data to obtain an emotion feature vector; performing probabilistic classification processing according to the emotion feature vector to obtain a multi-dimensional emotion probability distribution; performing scene constraint correction on the multi-dimensional emotion probability distribution to obtain a corrected emotion probability distribution; calculating confidence residual before and after correction and performing amplitude convergence calculation to obtain a comprehensive emotion intensity value; and when the comprehensive emotion intensity value meets a preset threshold value, generating an emotion early warning result according to the corrected emotion probability distribution and the comprehensive emotion intensity value. The method can realize non-contact, sleep scene adaptive and refined emotion perception and early warning.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition and human-computer interaction technology, and in particular to an emotion perception method and system based on millimeter-wave radar. Background Technology

[0002] Currently, emotion perception technology has wide-ranging applications in areas such as mental health monitoring, smart elderly care, childcare, and smart homes. With increasing societal pressures, emotional health issues in home settings are receiving growing attention, and users have an urgent need for long-term emotion monitoring solutions that do not require wearing devices and do not infringe on privacy. Emotion recognition based on physiological signals has become an important research direction in this field in recent years. Its core principle lies in the close correlation between autonomic nervous system activity and emotional state. For example, physiological indicators such as heart rate, respiratory rate, and heart rate variability (HRV) change accordingly with emotional fluctuations. By collecting and analyzing these objective physiological parameters, indirect inference of emotional state can be achieved.

[0003] In one existing technology, an emotion monitoring solution based on smart sensors collects the user's electrocardiogram (ECG) signals and heart rate data through wearable devices such as smart bracelets or chest straps. Physiological parameters such as HRV are extracted and compared with preset fixed thresholds. When a parameter exceeds the threshold range, it is determined to be a specific emotion category. The determination result is transmitted via Bluetooth to a mobile application for display and text reminders. However, in practical applications, this solution suffers from several drawbacks. Wearable devices require active user wearing and good contact, leading to poor compliance in scenarios such as home sleep and long-term continuous monitoring, making truly imperceptible emotion monitoring impossible. Furthermore, the determination rules are overly simplistic, relying solely on threshold comparisons of a single indicator, lacking temporal correlation fusion of multiple parameters such as heart rate, respiratory rate, and body movement amplitude, and failing to output emotion probability distribution and confidence information, making it difficult to support refined intervention decisions.

[0004] In summary, existing technologies cannot achieve refined emotion perception through non-contact, multi-parameter fusion. Summary of the Invention

[0005] This invention provides a method and system for emotion perception based on millimeter-wave radar to solve the problems of unstable quality and difficulty in ensuring semantic coherence in the generation of archive summaries.

[0006] Firstly, in order to solve the above-mentioned technical problems, the present invention provides an emotion perception method based on millimeter-wave radar, comprising: Acquire raw radar data and sleep state context, and demodulate the raw radar data using baseband processing to obtain basic time series data; The basic time series data is subjected to independent sliding window mean calculation to obtain a time series smoothed sequence, and multi-parameter time series feature extraction is performed based on the time series smoothed sequence to obtain a sentiment feature vector; Based on the emotional feature vector, probabilistic emotion classification is performed to obtain a multidimensional emotion probability distribution. The multidimensional emotion probability distribution is modified by scene constraints using the sleep state context to obtain the modified emotion probability distribution. Calculate the maximum confidence residual between the multidimensional emotion probability distribution and the modified emotion probability distribution, and perform amplitude convergence calculation based on the maximum confidence residual to obtain the comprehensive emotion intensity value; When the overall emotion intensity value meets the preset high arousal negative threshold, an emotion warning result is generated based on the modified emotion probability distribution and the overall emotion intensity value.

[0007] In a second aspect, the present invention provides an emotion perception system based on millimeter-wave radar, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0008] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0009] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention uses a 60GHz millimeter-wave radar module to collect multi-dimensional physiological parameters such as human heart rate, respiratory rate, heart rate waveform, respiratory waveform, and body movement amplitude in a non-contact manner, and outputs them to the ESP32 main control chip through a UART interface. This solution can continuously collect physiological signals in scenarios such as sleeping or sitting at home without the user wearing any electrodes or devices, thereby achieving truly imperceptible long-term emotion monitoring and overcoming the technical defects of wearable devices that lead to poor user compliance due to wearing discomfort.

[0010] (2) This invention utilizes a self-developed emotion analysis algorithm built into the main control chip to extract temporal features and perform sliding window smoothing on multiple parameters such as heart rate, respiratory rate, HRV estimation, and body movement amplitude output by the radar. The probability distributions of nine emotion categories are calculated using the Sigmoid and Gaussian functions, respectively. After normalization, the probability values, confidence levels, and primary and secondary emotion categories for each category are output. This scheme replaces threshold comparison of a single indicator with multi-parameter temporal correlation fusion and outputs a complete probability vector instead of a single judgment result, thereby achieving refined emotion perception and providing a confidence basis for tiered intervention decisions.

[0011] (3) This invention constructs a structured Prompt from the real-time emotion analysis results, calls a large language model to generate personalized voice response text, synthesizes it using TTS, and drives a speaker to broadcast it via an I2S interface. When severe negative emotions persist for more than a preset duration, the system further collects the user's voice through a microphone, converts it to text using ASR, and conducts multiple rounds of dialogue with the large language model. After the dialogue ends, the intervention log is automatically recorded. This solution uses a large language model to replace the playback method of a pre-stored audio library, making the intervention content personalized, context-aware, and coherent. At the same time, it forms a complete "perception-analysis-intervention-feedback" closed loop through two-way voice dialogue, thereby realizing immediate emotion guidance and intervention. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of an emotion perception method based on millimeter-wave radar provided in the first embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Reference Figure 1 The first embodiment of the present invention provides an emotion perception method based on millimeter-wave radar, comprising the following steps: S11, acquire the radar raw data and sleep state context, and demodulate the radar raw data into baseband data to obtain basic timing data; S12, perform independent sliding window mean calculation on the basic time series data to obtain a time series smoothed sequence, and perform multi-parameter time series feature extraction based on the time series smoothed sequence to obtain an emotion feature vector; S13, perform probabilistic emotion classification based on the emotion feature vector to obtain a multidimensional emotion probability distribution; S14, use the sleep state context to perform scene constraint correction on the multidimensional emotion probability distribution to obtain the corrected emotion probability distribution; S15, calculate the maximum confidence residual between the multidimensional emotion probability distribution and the modified emotion probability distribution, and perform amplitude convergence calculation based on the maximum confidence residual to obtain the comprehensive emotion intensity value; S16, when the comprehensive emotion intensity value meets the preset high arousal negative threshold, an emotion warning result is generated based on the modified emotion probability distribution and the comprehensive emotion intensity value.

[0015] In step S11, the radar raw data and sleep state context are acquired, and the radar raw data is demodulated and processed to obtain basic timing data.

[0016] The raw radar data is demodulated and processed to obtain basic time-series data, including: The raw radar data is mixed to obtain an intermediate frequency signal; The intermediate frequency signal is subjected to analog-to-digital conversion to obtain a digital intermediate frequency signal; The digital intermediate frequency signal is subjected to frequency domain demodulation processing to obtain basic timing data.

[0017] In this embodiment, the raw radar data refers to the structured data frames output by the R60ABD1 millimeter-wave radar module through its UART interface. This radar module integrates a radio frequency antenna, a radar chip, and a high-speed MCU, internally handling all low-level signal processing from radio frequency transmission and reception to vital sign parameter extraction. The radar module's output data frames include a frame header, control word, command word, length identifier, data segment, and frame trailer. The radar module outputs structured data frames at a baud rate of 115200bps via the UART interface. In this embodiment, the radar module outputs data frames at a frequency of 1 frame per second, with a data frame format of 8 data bits, 1 stop bit, and no parity bit. The definitions of each function point in the data segment are specified by the radar module's datasheet. In this embodiment, the heart rate value (DP6) occupies two bytes, with units of beats per minute; the respiratory rate value (DP8) occupies two bytes, with units of beats per minute; the heart rate waveform (DP7) occupies five bytes, representing the original heart rate waveform amplitude sequence; the respiratory waveform (DP10) occupies five bytes, representing the original respiratory waveform amplitude sequence; body movement amplitude (DP4) occupies one byte, with a value range of 0 to 100, representing the intensity of body movement at the current moment; and the bed-entry / bed-exit status (DP11) occupies one byte, where a value of 1 indicates someone is in a bed-exit state, a value of 2 indicates someone is in a bed-resting state, and a value of 0 indicates no one is present. The data is acquired by continuously receiving data frames output by the radar module through the UART receive pin (RXD0) of the ESP32-WROVER-E-N16R8 main control chip. After receiving each complete frame, each field is parsed sequentially according to the above frame structure. After successful verification, the values ​​corresponding to the above function points are extracted from the data segment.

[0018] The raw radar data undergoes frequency mixing to obtain an intermediate frequency (IF) signal. This mixing is performed by a mixer within the R60ABD1 radar module. The radar module transmits a 60GHz linear frequency modulated continuous wave signal. This signal is reflected by the human chest cavity and captured by the radar module's receiving antenna. The received and transmitted signals are mixed in the mixer. Due to a time delay between the transmitted and received signals (proportional to the distance between the target and the radar), there is a frequency difference between them. The mixing produces a new signal component containing this frequency difference. After filtering out the high-frequency component using a low-pass filter, the remaining difference frequency signal is the IF signal.

[0019] The intermediate frequency (IF) signal is subjected to analog-to-digital (ADC) conversion to obtain a digital IF signal. This ADC is performed by an integrated ADC within the radar module. Under the control of a sampling clock, this ADC samples the instantaneous voltage value of the analog IF signal at equal time intervals, converting the continuous-time analog IF signal into a discrete digital sampling sequence, thus forming the digital IF signal.

[0020] The digital intermediate frequency (IF) signal is demodulated in the frequency domain to obtain basic time-series data. The frequency domain demodulation process includes two sub-steps: range gate localization and phase extraction. In the range gate localization sub-step, a Fast Fourier Transform (FFT) is performed on each frame of the digital IF signal to transform the time-domain sampling sequence to the frequency domain, obtaining the frequency domain spectrum. The complex modulus value corresponding to each frequency index in the frequency domain spectrum represents the reflected energy intensity of the target at that range gate. The frequency index with the largest modulus value corresponds to the range gate where the target is located, and this index number is determined as the target range gate index.

[0021] In the phase extraction sub-step, for the frequency domain spectrum of each frame, a complex number at the target distance gate index position is extracted. This complex number contains a real part and an imaginary part. The phase value corresponding to the signal of that frame is calculated based on the real and imaginary parts. After processing multiple frames of signals continuously, the phase values ​​of each frame are arranged in chronological order to obtain the original phase sequence. Phase unwinding processing is performed on the original phase sequence. Phase unwinding adopts a frame-by-frame comparison and correction method. Starting from the second frame, the difference between the phase value of the current frame and the phase value of the previous frame is calculated frame by frame. If the difference is greater than π, 2π is subtracted from the current frame and subsequent frames. If the difference is less than -π, 2π is added to obtain a continuous phase sequence without entanglement. The phase values ​​of each frame in the unwinding phase sequence are converted into displacement values ​​to obtain a time series characterizing the micro-displacement of the thoracic cavity. This sequence is the basic time series data.

[0022] In this embodiment, the sleep state context is obtained as follows. The ESP32 main control chip internally runs a sleep analysis module. This module takes the body movement amplitude (DP4), heart rate value (DP6), and respiratory rate value (DP8) output by the radar module as input and uses a dual-timescale body movement smoothing method to determine the sleep state. The fast body movement smoothing value is calculated as follows: the fast body movement smoothing value of the current frame equals 0.3 multiplied by the original body movement amplitude value of the current frame plus 0.7 multiplied by the fast body movement smoothing value of the previous frame. The slow body movement smoothing value is calculated as follows: the slow body movement smoothing value of the current frame equals 0.01 multiplied by the original body movement amplitude value of the current frame plus 0.99 multiplied by the slow body movement smoothing value of the previous frame.

[0023] In this embodiment, the fast smoothing coefficient of 0.3 and the slow smoothing coefficient of 0.01 are set based on the fact that the physical meaning of the smoothing coefficient is the weight of the current measurement value in the smoothing result. A larger fast smoothing coefficient (0.3) allows the smoothing result to respond quickly to sudden changes in body movement, thus capturing short-term activity; a smaller slow smoothing coefficient (0.01) makes the smoothing result almost unaffected by a single body movement, reflecting long-term resting trends. These two coefficients fall within the conventional value range in the art; the fast smoothing coefficient is typically between 0.2 and 0.5, and the slow smoothing coefficient is typically between 0.005 and 0.02. In this embodiment, they are set to 0.3 and 0.01, respectively.

[0024] When the slow body movement smoothing value is below the sleep onset threshold, the heart rate is below the user's wakefulness baseline, and the respiratory rate is below the user's wakefulness baseline, the user is determined to have entered a sleep state. After entering a sleep state, when the slow body movement smoothing value is below 50% of the sleep onset threshold, the heart rate is below 90% of the user's average heart rate during sleep, and this lasts for more than five minutes, it is determined to be a deep sleep stage. When the slow body movement smoothing value is below the sleep onset threshold but above 20% of the body movement level during deep sleep, and the heart rate rises back to the range of 95% to 105% of the user's average heart rate during sleep, it is determined to be a REM sleep stage. If any of the above stage determination conditions are not met, it is determined to be a light sleep stage. The sleep analysis module stores the current sleep stage in a numerical encoding form in the ESP32's memory variable, where 0 represents wakefulness, 1 represents light sleep, 2 represents deep sleep, and 3 represents REM sleep. During step S11, the current sleep stage encoding value is read from this memory variable, which is the sleep state context. The sleep threshold is a preset value, which is determined during the system initialization phase by collecting the slow body motion smoothing value of the user in a conscious and quiet sitting state. The collection time is five minutes, and the arithmetic mean of the slow body motion smoothing value within this period is taken as the sleep threshold.

[0025] In step S12, the basic time series data is subjected to independent sliding window mean calculation to obtain a time series smoothed sequence, and multi-parameter time series feature extraction is performed based on the time series smoothed sequence to obtain an emotion feature vector.

[0026] Specifically, multi-parameter temporal feature extraction is performed based on the temporal smoothing sequence to obtain an emotion feature vector, including: The time series data is subjected to sliding window mean calculation to obtain a time series smoothed sequence; Heart rate variability, respiratory rate standard deviation, and kinetic energy values ​​were extracted from the time-series smoothed sequence. The heart rate variability, respiratory rate standard deviation, and kinetic energy value are concatenated into a vector to obtain an emotion feature vector.

[0027] In this embodiment, the basic time-series data refers to the time series representing the micro-displacement of the thoracic cavity output in step S11. Since the basic time-series data output by the radar module contains a composite displacement signal superimposed from respiratory and cardiac movements, to facilitate subsequent physiological parameter calculations, the basic time-series data is first filtered and separated, converting it into heart rate and respiratory rate sequences. Specifically, a bandpass filter is used to filter the basic time-series data, extracting the respiratory frequency band signal and the heart rate frequency band signal, where the respiratory frequency band is 0.1 Hz to 0.5 Hz and the heart rate frequency band is 0.8 Hz to 2.5 Hz. After filtering, time-domain analysis is performed on the respiratory and heart rate frequency band signals respectively, extracting the signal period within each time window and converting it into respiratory rate and heart rate values, in units of breaths per minute. Simultaneously, the body motion amplitude value output by the radar module is directly used as the body motion amplitude value at that moment. Thus, the basic time-series data is converted into a heart rate sequence, respiratory rate sequence, and body motion amplitude sequence with a fixed time interval as the time axis.

[0028] A sliding window mean calculation is performed on the aforementioned basic time-series data to obtain a time-smoothed sequence. The sliding window mean calculation refers to performing time-domain smoothing on the heart rate sequence, respiratory rate sequence, and body movement amplitude sequence respectively. The length of the sliding window is a preset value. In this embodiment, the sliding window length is set to 15 sampling points. This window length is set based on the fact that the purpose of the sliding window mean calculation is to filter out high-frequency random fluctuations in physiological signals while retaining low-frequency trend components caused by emotional changes. A 15-second window can cover at least 2 to 3 complete respiratory cycles and multiple heartbeat cycles, which is sufficient to smooth the periodic fluctuations caused by breathing and heartbeats. At the same time, the window length is moderate and will not over-smooth the trends of emotion-related vital signs.

[0029] For the heart rate sequence, starting from the 15th second, the heart rate values ​​from the 1st to the 15th second are summed and divided by 15 to obtain the smoothed heart rate value corresponding to the 15th second. Then, the window is moved forward by 1 second, and the heart rate values ​​from the 2nd to the 16th second are summed and divided by 15 to obtain the smoothed heart rate value corresponding to the 16th second. This process is repeated until the entire heart rate sequence is traversed, resulting in the smoothed heart rate sequence. The sliding window mean calculation method for the respiratory rate sequence and body motion amplitude sequence is the same as for the heart rate sequence, yielding the smoothed respiratory rate sequence and the smoothed body motion sequence, respectively. These three smoothed sequences together constitute the temporal smoothing sequence of this step.

[0030] Heart rate variability, respiratory rate standard deviation, and kinetic energy values ​​are extracted from the smoothed time series. Heart rate variability refers to the rate of change of heart rate over time, used to characterize the instantaneous activation level of the autonomic nervous system. Specifically, taking the current moment as the endpoint, the smoothed heart rate values ​​of the first 10 seconds of the smoothed time series are extracted. A linear regression method is used to calculate the slope of heart rate change over time; this slope is the heart rate variability, measured in beats per minute per second. The 10-second window is set because the heart rate variability reflects the short-term trend of heart rate change. A window that is too long will smooth the trend, while a window that is too short is easily affected by fluctuations in a single heartbeat. A 10-second window is commonly used in this field for calculating short-term trends in heart rate variability analysis.

[0031] The standard deviation of respiratory rate refers to the dispersion of the smoothed respiratory rate values ​​within a 60-second window prior to the current moment relative to the mean respiratory rate within that window. It is used to characterize the regularity of respiratory rhythm. Specifically, the smoothed respiratory rate values ​​within the 60 seconds prior to the current moment are extracted from the time-series smoothed sequence, and the standard deviation of these 60 values ​​is calculated to obtain the respiratory rate standard deviation, expressed in breaths per minute. The 60-second window is set based on the fact that respiratory rate exhibits a slow change characteristic during emotional fluctuations, and the 60-second window can cover multiple respiratory cycles, ensuring the representativeness of the standard deviation statistics.

[0032] The body motion energy value refers to the cumulative energy of smoothed body motion values ​​within a 60-second window preceding the current moment, used to characterize the overall intensity of a user's physical activity. Specifically, the smoothed body motion values ​​for the 60 seconds preceding the current moment are extracted from the time-series smoothed sequence. Each smoothed body motion value is squared, and all squared values ​​are summed to obtain the body motion energy value. The body motion energy value represents the accumulation of the square of the body motion amplitude over the time window, characterizing the total intensity of physical activity.

[0033] The heart rate variability, respiratory rate standard deviation, and kinetic energy value are concatenated as vectors to obtain the emotion feature vector. Vector concatenation refers to combining three scalar values ​​into a multi-dimensional vector in a fixed order. Specifically, the heart rate variability value is used as the first dimension of the emotion feature vector, the respiratory rate standard deviation value as the second dimension, and the kinetic energy value as the third dimension, organized into a three-dimensional vector in the form of (heart rate variability, respiratory rate standard deviation, kinetic energy value). This three-dimensional vector is the emotion feature vector, which will be used as input for subsequent emotion classification processing.

[0034] In step S13, probabilistic emotion classification is performed based on the emotion feature vector to obtain a multidimensional emotion probability distribution, including: Calculate the activation probability of each emotion based on the emotion feature vector; Calculate the deviation between the emotion feature vector and the preset typical feature centers of various emotions; The activation probability and the deviation are fused together to obtain the probability values ​​corresponding to each type of emotion; The probability values ​​corresponding to the various emotions are normalized to obtain a multidimensional emotion probability distribution.

[0035] In this embodiment, the emotion feature vector refers to the three-dimensional vector output in step S12, which includes three dimensions: heart rate variability, respiratory rate standard deviation, and kinetic energy value. The goal of probabilistic emotion classification is to map this three-dimensional vector into probability values ​​for nine emotion categories. These nine emotion categories include calm, happiness, excitement, anxiety, anger, sadness, stress, relaxation, and unknown. All emotion categories share the same set of input features but have their own independent classification parameters.

[0036] The activation probability of each emotion is calculated based on the emotional feature vector. Specifically, for each of the nine emotion categories, an independent set of weight coefficients and bias terms for a sigmoid function is configured. The sigmoid function is a well-known classification function, and its output value is between 0 and 1, representing the activation level of the emotion category under the current feature, i.e., the activation probability. The weight coefficients of each emotion are set using an equal-weighting method, that is, the weights of heart rate variability, respiratory rate standard deviation, and kinetic energy value are all equal, and the bias terms are all set to zero.

[0037] The deviation between the emotional feature vector and the preset typical feature centers for each emotion category is calculated. For each of the nine emotion categories, a typical feature center is preset. This center is a three-dimensional vector representing the typical position of that emotion in the physical characteristic space. The typical feature centers are constructed by using the center values ​​of the typical value ranges of the nine emotions in the three dimensions of heart rate variability, respiratory rate standard deviation, and body energy value as the three-dimensional coordinate values ​​of the typical feature center for that emotion category. In this embodiment, the values ​​of the typical feature centers for each emotion category are based on the typical physical manifestations of each emotion category known in the field of emotional physiology. All the above-mentioned dimension values ​​are dimensionless normalized values. The values ​​of the typical feature centers for each emotion category are as follows: calm (-0.1, 1.0, 50), happy (0.3, 1.5, 120), excited (0.8, 2.0, 200), anxious (0.6, 2.5, 80), angry (1.0, 2.8, 180), sad (-0.2, 0.8, 30), stressed (0.5, 2.2, 100), relaxed (-0.3, 0.6, 40), and unknown (0, 0, 0). The values ​​of the aforementioned typical feature centers are based on the typical physiological manifestations of each emotion category across three physiological dimensions: heart rate variability, respiratory rate standard deviation, and kinetic energy value. Calm corresponds to a stable heart rate, regular breathing, and minimal kinetic energy; happiness corresponds to a slightly elevated heart rate, slightly faster breathing, and moderate kinetic energy; excitement corresponds to a rapidly rising heart rate, faster breathing, and more kinetic energy; anxiety corresponds to a relatively fast and unstable heart rate, irregular breathing, and minimal kinetic energy; anger corresponds to a sharply rising heart rate, extremely irregular breathing, and more kinetic energy; sadness corresponds to a relatively low heart rate, slow and irregular breathing, and very little kinetic energy; stress corresponds to a persistently high heart rate, irregular breathing, and moderate kinetic energy; relaxation corresponds to a low heart rate, slow and regular breathing, and minimal kinetic energy; and "unknown" corresponds to the median value for each dimension. For the currently input emotion feature vector, the distance between it and the nine typical feature centers is calculated. A larger distance indicates a greater deviation from the typical feature center of that emotion, while a smaller distance indicates a closer similarity.

[0038] The activation probability and the deviation are fused to obtain the probability values ​​for each emotion category. For each of the nine emotion categories, the activation probability and deviation are fused to obtain the original probability value for that emotion category. Specifically, the deviation is subtracted from the activation probability to obtain the difference; when the difference is greater than 0, this difference is used as the original probability value for that emotion category; when the difference is less than or equal to 0, the original probability value for that emotion category is set to 0. This calculation method means that only when the activation probability of an emotion category is greater than its deviation is a positive probability value assigned to that emotion; if the activation probability is less than or equal to the deviation, it means that although the current feature is activated, it deviates far from the typical features of that emotion and is not assigned a probability value. The original probability values ​​for each of the nine emotion categories are calculated separately.

[0039] The probability values ​​corresponding to each emotion category are normalized to obtain a multidimensional emotion probability distribution. The original probability values ​​of the nine emotion categories are summed to obtain the total probability value. For each emotion category, its original probability value is divided by the total probability value to obtain a normalized probability value. The sum of the nine normalized probability values ​​equals 1, and each probability value is between 0 and 1. Subsequently, the normalized probability distribution is post-processed. The maximum value among the nine normalized probability values ​​is found. If the maximum value is greater than 0.75, it is multiplied by 1.3 to amplify it, while the other eight probability values ​​are proportionally reduced to ensure the sum remains 1. If the maximum value is less than 0.2, the emotion category corresponding to the maximum probability value is forcibly labeled as "unknown," and the probability value corresponding to that category is set to 1, while the other eight categories are set to 0. After post-processing, the probability values ​​corresponding to each of the nine emotion categories together constitute the multidimensional emotion probability distribution output in this step. The multidimensional emotion probability distribution can be represented as a probability vector, where each dimension corresponds to the probability value of a type of emotion. The mapping relationship between each dimension index and the emotion category is in a fixed order: dimension 1 is calm, dimension 2 is happy, dimension 3 is excited, dimension 4 is anxious, dimension 5 is angry, dimension 6 is sad, dimension 7 is stressed, dimension 8 is relaxed, and dimension 9 is unknown.

[0040] In step S14, the multidimensional emotion probability distribution is modified by scene constraints using the sleep state context to obtain a modified emotion probability distribution, including: When the sleep state context indicates that the current stage is deep sleep, the emotion categories with probability values ​​greater than a preset threshold in the multidimensional emotion probability distribution are suppressed, and the relaxation and calm emotions are enhanced to obtain the deep sleep correction probability value. When the sleep state context indicates that the current stage is REM sleep, the confidence decay processing is performed on the multidimensional emotion probability distribution to obtain the REM corrected probability value. A modified mood probability distribution is generated based on the deep sleep correction probability value or the rapid eye movement correction probability value.

[0041] In this embodiment, the sleep state context refers to the current sleep stage encoding value obtained in step S11, which takes values ​​of 0, 1, 2, or 3, corresponding to wakefulness, light sleep, deep sleep, and REM sleep, respectively. The multidimensional emotion probability distribution refers to the probability value vectors corresponding to the nine emotion categories output in step S13. The basis for scene constraint correction is that when the user is asleep, physiological signals mainly reflect the sleep physiological state rather than the emotional state. If the emotion classification rules of the awake state are directly used for judgment, normal physiological fluctuations during sleep will be misjudged as negative emotions. Therefore, it is necessary to make targeted corrections to the multidimensional emotion probability distribution according to the sleep stage.

[0042] When the sleep state context indicates that the current stage is deep sleep, the emotion categories with probability values ​​greater than a preset threshold in the multidimensional emotion probability distribution are suppressed, while relaxation and calmness are enhanced to obtain a deep sleep corrected probability value. The physiological characteristics of deep sleep include a heart rate below baseline, regular and stable breathing, and minimal body movement. The physiological signals of this stage are similar in signs to the "calm" and "relaxed" states during wakefulness, but physiological fluctuations during sleep should not be interpreted as emotional signals. The suppression process involves iterating through the probability values ​​corresponding to each of the nine emotion categories in the multidimensional emotion probability distribution, identifying emotion categories with probability values ​​greater than a preset threshold as high-probability emotion categories. The preset threshold is a system preset value, which can be set by those skilled in the art according to the actual application scenario, and is typically 0.5. The probability value corresponding to the high-probability emotion category (excluding relaxation and calmness) is multiplied by a preset first suppression coefficient, thus reducing the probability value of that emotion.

[0043] The enhancement of relaxation and calmness is achieved by multiplying the probability value corresponding to relaxation by a preset first enhancement coefficient, and the probability value corresponding to calmness by the same preset first enhancement coefficient. The first inhibition coefficient ranges from 0 to 1, and in this embodiment, the first inhibition coefficient is 0.2. The first enhancement coefficient ranges from greater than 1, and in this embodiment, the first enhancement coefficient is 1.5. 0.2 and 1.5 are conventional values ​​in the art. The above coefficients are chosen based on the fact that during deep sleep, the confidence level of emotion judgment should be significantly reduced, and high-probability emotion categories are suppressed to a lower level; at the same time, relaxation and calmness, as the emotion labels closest to the sleep state, should be moderately enhanced to reflect the physiological state of sleep, but not so that they absolutely dominate and completely suppress the possibility of other emotions. After completing the above inhibition and enhancement processing, the nine processed probability values ​​are normalized, that is, the nine probability values ​​are summed to obtain the total probability value, and each probability value is divided by the total probability value, so that the sum of the nine probability values ​​equals 1. The nine normalized probability values ​​together constitute the deep sleep correction probability value.

[0044] It should be noted that if there are no emotion categories with a probability value greater than the preset threshold in the multidimensional emotion probability distribution (i.e., the probability values ​​of all emotion categories are lower than the preset threshold, indicating that the confidence of the current emotion judgment is low), then there is no need to perform suppression processing. Only relaxation and calm emotions are enhanced, and then normalized to obtain the deep sleep corrected probability value.

[0045] When the sleep context indicates that the current stage is REM sleep, confidence attenuation processing is performed on the multidimensional emotion probability distribution to obtain a REM-corrected probability value. The physiological characteristics of REM sleep include increased heart rate fluctuations, irregular breathing, and increased body movement frequency compared to deep sleep. These physiological signals often resemble the characteristics of negative emotions such as anxiety and stress when awake, but these physiological fluctuations during REM sleep are normal sleep physiological phenomena and should not be interpreted as emotional signals. Confidence attenuation processing involves extracting the maximum probability value and its corresponding emotion category from the multidimensional emotion probability distribution; multiplying the maximum probability value by a preset second inhibition coefficient, i.e., reducing the maximum probability value, thereby reducing the confidence of the dominant emotion category. The preset second inhibition coefficient ranges from 0 to 1 (excluding 0 and 1), and in this embodiment, the second inhibition coefficient is 0.5, meaning the maximum probability value is halved. This value is based on the fact that physiological signal fluctuations are large during REM sleep, but emotional information should not be completely eliminated; therefore, halving the confidence reduces the risk of misjudgment while retaining some emotional distinguishability. The other eight probability values ​​remain unchanged. The values ​​are then normalized so that the sum of the nine probability values ​​equals 1. These nine normalized probability values ​​together constitute the fast eye movement correction probability value.

[0046] A modified emotion probability distribution is generated based on the deep sleep modified probability value or the REM sleep modified probability value. Since deep sleep and REM sleep are physiologically mutually exclusive, a user cannot be in both sleep stages simultaneously. Therefore, step S14 triggers at most one of the two branches based on the current sleep state context. If the user is currently in deep sleep, the deep sleep modified probability value is the modified emotion probability distribution; if the user is currently in REM sleep, the REM modified probability value is the modified emotion probability distribution; if the user is currently awake or in light sleep, no modification is required, and the multidimensional emotion probability distribution output in step S13 is directly used as the modified emotion probability distribution. The modified emotion probability distribution is output as a 9-dimensional probability vector to subsequent steps. The mapping relationship between each dimension index and emotion category is in a fixed order: dimension 1: calm; dimension 2: happy; dimension 3: excited; dimension 4: anxious; dimension 5: angry; dimension 6: sad; dimension 7: stressed; dimension 8: relaxed; dimension 9: unknown.

[0047] In step S15, the maximum confidence residual between the multidimensional emotion probability distribution and the modified emotion probability distribution is calculated, and amplitude convergence calculation is performed based on the maximum confidence residual to obtain the comprehensive emotion intensity value.

[0048] The comprehensive emotion intensity value is obtained by performing amplitude convergence calculation based on the maximum confidence residual, including: The initial intensity value is obtained by performing a nonlinear transformation on the maximum confidence residual; The initial intensity value is normalized to obtain the comprehensive emotion intensity value.

[0049] In this embodiment, the multidimensional emotion probability distribution refers to the 9-dimensional probability vector output in step S13 without sleep constraint correction, and the corrected emotion probability distribution refers to the 9-dimensional probability vector output in step S14 after sleep constraint correction. The maximum confidence residual is calculated by extracting the maximum probability values ​​from both the multidimensional and corrected emotion probability distributions, subtracting the maximum probability value of the corrected emotion probability distribution from the maximum probability value of the multidimensional emotion probability distribution, and taking the absolute value of the difference. The physical meaning of this residual is the adjustment magnitude of the dominant emotion confidence level by sleep constraint correction, reflecting the degree to which the user's current emotional state is interfered with by the sleep physiological state. The calculation of the maximum confidence residual is completed in the initial stage of step S15 and serves as the input for subsequent magnitude convergence calculations.

[0050] The amplitude convergence calculation is performed based on the maximum confidence residual to obtain the comprehensive emotion intensity value. A nonlinear transformation is then applied to the maximum confidence residual to obtain the initial intensity value. The nonlinear transformation uses a sigmoid function to map the maximum confidence residual from its original range to a continuous range between 0 and 1. The sigmoid function is a well-known function whose curve exhibits high sensitivity when the independent variable's value is in the middle range and tends to saturate when the independent variable's value is at both ends. This characteristic is suitable for mapping confidence residuals to emotion intensity values. The maximum confidence residual is substituted into the sigmoid function as the independent variable to calculate the function output value, which is between 0 and 1, representing the initial intensity value. The proportional gain of the sigmoid function is set to 4.0, and the bias coefficient is set to 0.5 to ensure that the intensity value has reasonable discriminative power within the typical confidence residual range.

[0051] The initial intensity value is normalized to obtain the comprehensive emotion intensity value. Since the initial intensity value is already within the range of 0 to 1, the maximum probability value in the corrected emotion probability distribution is obtained; the initial intensity value is multiplied by this maximum probability value to obtain the comprehensive emotion intensity value. This calculation method means that the higher the confidence level of the dominant emotion after sleep constraint correction, the higher the final comprehensive emotion intensity value; conversely, if the probability distribution of each emotion tends to be uniform after correction (lower maximum probability value), the comprehensive emotion intensity value will decrease accordingly. If the comprehensive emotion intensity value exceeds the upper limit of 1.0 after calculation in the above manner, it is forcibly truncated to 1.0. If the value is lower than the lower limit of 0, it is forcibly truncated to 0. The comprehensive emotion intensity value represents the overall arousal level and intensity of the user's current emotional state; the higher the value, the stronger the physiological arousal of the emotion. In specific applications, the comprehensive emotion intensity value is used to compare with a preset high arousal negative threshold to determine whether an emotion warning result needs to be generated.

[0052] In step S16, when the comprehensive emotion intensity value meets a preset high arousal negative threshold, an emotion warning result is generated based on the modified emotion probability distribution and the comprehensive emotion intensity value, including: If the overall emotion intensity value is greater than the preset high arousal negative threshold, then the emotion category with the highest probability value is extracted from the modified emotion probability distribution as the target emotion category. The target emotion category and the comprehensive emotion intensity value are encapsulated to obtain the emotion warning result.

[0053] In this embodiment, the comprehensive emotion intensity value refers to the numerical value output in step S15, which characterizes the overall arousal level and intensity of the user's current emotional state, and its value ranges from 0 to 1. The corrected emotion probability distribution refers to the 9-dimensional probability vector output in step S14, with each dimension corresponding to the probability values ​​of nine emotion categories: calm, happy, excited, anxious, angry, sad, stressed, relaxed, and unknown. The high arousal negative threshold is a critical value used to determine whether the current emotion requires triggering an alert, and it is a system preset value. The high arousal negative threshold is determined by its ability to distinguish between negative emotional states requiring alerts and other states that do not require alerts. In the field of emotion recognition, an arousal level higher than 0.6 is generally considered a high arousal state; in this embodiment, the high arousal negative threshold is set to 0.65.

[0054] If the overall emotional intensity value is greater than the preset high arousal negative threshold, an early warning condition is triggered. The overall emotional intensity value is compared with 0.65. If the overall emotional intensity value is greater than 0.65, the current emotional state is determined to meet the high arousal negative condition, and the early warning process is initiated. If the overall emotional intensity value is less than or equal to 0.65, the current situation is determined not to meet the early warning condition, and no further operations are performed. The process returns to step S11 to continue collecting the next round of data.

[0055] The emotion category with the highest probability value is extracted from the modified emotion probability distribution as the target emotion category. The nine probability values ​​in the modified emotion probability distribution are traversed, and the current highest probability value and its corresponding dimension index are recorded by comparison. After traversal, the dimension index is mapped to the corresponding emotion category name. The mapping relationship between dimension index and emotion category is: 1st dimension: calm; 2nd dimension: happy; 3rd dimension: excited; 4th dimension: anxious; 5th dimension: angry; 6th dimension: sad; 7th dimension: stressed; 8th dimension: relaxed; 9th dimension: unknown. The extracted target emotion category is one of the above nine categories, used to indicate the dominant category of the current emotion.

[0056] The target emotion category and the comprehensive emotion intensity value are encapsulated to obtain an emotion warning result. Data encapsulation refers to combining the target emotion category and the comprehensive emotion intensity value into a single warning record. This warning record can be uploaded to a cloud platform via Wi-Fi, and the cloud platform will push a notification to the user's terminal according to the warning rules. If the user is in a deep sleep stage when the warning is triggered, no voice intervention will be initiated while outputting the warning result; only the warning log will be recorded.

[0057] In summary, this invention uses millimeter-wave radar to non-contactly collect multi-dimensional physiological parameters such as heart rate and respiratory rate, combines multi-parameter time-series fusion with nine types of probabilistic emotion analysis and sleep scenario constraint correction, and drives personalized voice intervention and two-way voice dialogue to achieve long-term emotion pattern analysis and health early warning.

[0058] The second embodiment of the present invention provides an emotion perception system based on millimeter-wave radar, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method described above.

[0059] It should be noted that the millimeter-wave radar-based emotion perception system provided in this embodiment of the invention is used to execute all the process steps of the millimeter-wave radar-based emotion perception method in the above embodiment. The working principles and beneficial effects of the two correspond one-to-one, so they will not be described again.

[0060] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0061] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for emotion perception based on millimeter-wave radar, characterized in that, include: Acquire raw radar data and sleep state context, and demodulate the raw radar data using baseband processing to obtain basic time series data; The basic time series data is subjected to independent sliding window mean calculation to obtain a time series smoothed sequence, and multi-parameter time series feature extraction is performed based on the time series smoothed sequence to obtain a sentiment feature vector; Based on the emotional feature vector, probabilistic emotion classification is performed to obtain a multidimensional emotion probability distribution. The multidimensional emotion probability distribution is modified by scene constraints using the sleep state context to obtain the modified emotion probability distribution. Calculate the maximum confidence residual between the multidimensional emotion probability distribution and the modified emotion probability distribution, and perform amplitude convergence calculation based on the maximum confidence residual to obtain the comprehensive emotion intensity value; When the overall emotion intensity value meets the preset high arousal negative threshold, an emotion warning result is generated based on the modified emotion probability distribution and the overall emotion intensity value.

2. The emotion perception method based on millimeter-wave radar according to claim 1, characterized in that, The demodulation of the raw radar data to obtain basic time-series data includes: The raw radar data is mixed to obtain an intermediate frequency signal; The intermediate frequency signal is subjected to analog-to-digital conversion to obtain a digital intermediate frequency signal; The digital intermediate frequency signal is subjected to frequency domain demodulation processing to obtain basic timing data.

3. The emotion perception method based on millimeter-wave radar according to claim 1, characterized in that, The step of extracting multi-parameter temporal features from the temporal smoothing sequence to obtain an emotion feature vector includes: Heart rate variability, respiratory rate standard deviation, and kinetic energy values ​​were extracted from the time-series smoothed sequence. The heart rate variability, respiratory rate standard deviation, and kinetic energy value are concatenated into a vector to obtain an emotion feature vector.

4. The emotion perception method based on millimeter-wave radar according to claim 1, characterized in that, The step of performing probabilistic emotion classification based on the emotion feature vector to obtain a multidimensional emotion probability distribution includes: Calculate the activation probability of each emotion based on the emotion feature vector; Calculate the deviation between the emotion feature vector and the preset typical feature centers of various emotions; The activation probability and the deviation are fused together to obtain the probability values ​​corresponding to each type of emotion; The probability values ​​corresponding to the various emotions are normalized to obtain a multidimensional emotion probability distribution.

5. The emotion perception method based on millimeter-wave radar according to claim 1, characterized in that, The step of using the sleep state context to perform scene constraint correction on the multidimensional emotion probability distribution to obtain a corrected emotion probability distribution includes: When the sleep state context indicates that the current stage is deep sleep, the emotion categories with probability values ​​greater than a preset threshold in the multidimensional emotion probability distribution are suppressed, and the relaxation and calm emotions are enhanced to obtain the deep sleep correction probability value. When the sleep state context indicates that the current stage is REM sleep, the confidence decay processing is performed on the multidimensional emotion probability distribution to obtain the REM corrected probability value. A modified mood probability distribution is generated based on the deep sleep correction probability value or the rapid eye movement correction probability value.

6. The emotion perception method based on millimeter-wave radar according to claim 1, characterized in that, The step of performing amplitude convergence calculation based on the maximum confidence residual to obtain the comprehensive emotion intensity value includes: The initial intensity value is obtained by performing a nonlinear transformation on the maximum confidence residual; The initial intensity value is normalized to obtain the comprehensive emotion intensity value.

7. The emotion perception method based on millimeter-wave radar according to claim 1, characterized in that, When the comprehensive emotion intensity value meets a preset high arousal negative threshold, generating an emotion warning result based on the modified emotion probability distribution and the comprehensive emotion intensity value includes: If the overall emotion intensity value is greater than the preset high arousal negative threshold, then the emotion category with the highest probability value is extracted from the modified emotion probability distribution as the target emotion category. The target emotion category and the comprehensive emotion intensity value are encapsulated to obtain the emotion warning result.

8. The emotion perception method based on millimeter-wave radar according to claim 1, characterized in that, The radar's raw data includes heart rate values, respiratory rate values, heart rate waveforms, respiratory waveforms, and body movement amplitude.

9. An emotion perception system based on millimeter-wave radar, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the method described in any one of claims 1 to 8.

Images