Emotion recognition method and robot based on millimeter wave radar

Abstract

The present invention relates to a millimeter-wave radar-based emotion recognition method and robot. The method comprises the following steps: receiving a millimeter-wave radar transmission signal and the corresponding echo signal, converting them into a point cloud format, removing static reflections, and obtaining dynamic point cloud data and timestamps of the surrounding environment; grouping the dynamic point cloud data into clusters; obtaining and tracking the target's center of mass estimate; transmitting a uniformly moving millimeter-wave signal horizontally to a target person, and receiving evenly spaced echo signals reflected by the target person; optimizing the point spread function and reconstructing a SAR image using a fast frequency-domain near-field imaging method; calculating the phase change of the human body signal, separating the target person's breathing and heartbeat signals; and extracting emotion-related features to obtain the target person's current emotion. Compared with existing technologies, the present invention can accurately identify the specific location of a person and achieve contactless, privacy-preserving emotion recognition over a long distance.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to an emotion recognition method and a robot based on millimeter wave radar. Background Art

[0002] With the advancement of technology, smart homes and the intelligent Internet of Things behind them are gradually demonstrating their irreplaceable advantages in the development of the elderly care industry. However, in addition to basic living needs, the spiritual needs of the elderly also need to be given sufficient care and attention.

[0003] Currently, emotional care for the elderly is still heavily reliant on manpower. Although some smart IoT devices can provide assistance by detecting basic physiological indicators of the human body, such as a wearable physiological indicator detection device disclosed in Chinese utility model patent CN219742686U, they are still imperfect. On the one hand, the detection of physiological indicators often requires the target to wear specific sensors, and often requires the target to be in a specific position or even maintain a specific posture, which can easily cause physical discomfort to the elderly and hinder all-day continuous detection; on the other hand, the analysis and interpretation of sensor data to the generation of feedback currently still relies largely on manpower, making it impossible to provide the elderly with appropriate feedback and emotional care in real time. The reliance on manpower also hinders all-day continuous care. Summary of the Invention

[0004] One of the purposes of the present invention is to provide an emotion recognition method and robot based on millimeter-wave radar. Based on the non-contact and high-resolution characteristics of millimeter-wave radar, a signal processing method is designed, and the corresponding emotions are obtained through machine learning, overcoming the problem of privacy exposure when using other technologies to identify emotions, and realizing contactless emotion recognition with privacy protection.

[0005] The second purpose of the present invention is to provide an emotion recognition method and robot based on millimeter-wave radar. By analyzing the rotational movement signal based on millimeter-wave radar, the problem that the user can only maintain a relatively fixed posture during emotion recognition in the original technology is solved, thereby being able to track the target position of the human body and achieve more flexible emotion recognition.

[0006] The third purpose of the present invention is to provide an emotion recognition method and robot based on millimeter-wave radar, and long-distance high-precision target recognition based on multi-aperture radar. The problem of reduced accuracy of millimeter-wave radar at long distances and reduced accuracy of emotion recognition is solved through point spread function (PSF) optimization and fast frequency domain near-field imaging method (RMA) algorithm, thereby obtaining a solution for long-distance and high-precision target emotion recognition.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] An emotion recognition method based on millimeter wave radar includes the following steps:

[0009] Receive the millimeter-wave radar's transmitted signal and the corresponding echo signal, mix them to obtain the intermediate frequency signal as the received data, convert the received data into a point cloud format, remove static reflections, and obtain dynamic point cloud data and timestamps of the surrounding environment;

[0010] Grouping the dynamic point cloud data into clusters;

[0011] Obtaining a target's center of mass estimate based on each cluster of dynamic point cloud data, tracking the center of mass estimate, and determining the target person's position at each moment;

[0012] Send a millimeter wave signal moving horizontally at a uniform speed to the target person, and receive the evenly spaced echo signals reflected by the target person;

[0013] performing point spread function optimization on the evenly spaced echo signals, and performing SAR image reconstruction and correction using a fast frequency domain near-field imaging method;

[0014] The phase change of the human body signal is calculated based on the reconstructed and corrected SAR image to obtain the phase difference signal, which is then separated to obtain the breathing and heartbeat signals of the target person.

[0015] Emotion-related features are extracted from the breathing and heartbeat signals, and the current emotion of the target person is identified based on the emotion-related features.

[0016] Furthermore, grouping the dynamic point cloud data into clusters specifically includes:

[0017] Using Mahalanobis distance as the metric, DBSCAN algorithm is applied to group dynamic point cloud data into clusters.

[0018] Furthermore, the process of obtaining the estimated centroid value includes:

[0019] Calculate the measurement noise covariance matrix corresponding to each cluster of point cloud data;

[0020] The centroid position is calculated based on the sum of the products of the inverse of the measurement noise covariance matrix and the measurement value of each point in each cluster point cloud data, and the noise covariance matrix of the centroid is obtained according to the inverse of the sum of the inverses of all measurement noise covariance matrices.

[0021] Furthermore, tracking the centroid estimate using a joint integrated probabilistic data association algorithm comprises the following steps:

[0022] Initialize the target state using the extended Kalman filter;

[0023] Use the joint integrated probabilistic data association algorithm to calculate the association probability and associate the new observation data with the existing trajectory;

[0024] Update the target's state estimate using an extended Kalman filter;

[0025] Determine whether to confirm or delete the track based on the probability of the target's existence.

[0026] Furthermore, the point spread function optimization specifically includes:

[0027] Designate a point target as a reference target in the imaging scene and calculate the round-trip propagation error between the reference target and the imaging target;

[0028] Based on the difference distribution of the round-trip propagation error between different point targets, a final point target is set, and phase error compensation is obtained based on the final point target phase history and the corresponding ideal quadratic phase history to obtain a signal after point spread function optimization.

[0029] Furthermore, the SAR image reconstruction correction specifically includes:

[0030] Performing a one-dimensional fast Fourier transform on each pulse echo in the signal after point spread function optimization to obtain a range spectrum, and performing a one-dimensional fast Fourier transform on each range spectrum to obtain an azimuth spectrum;

[0031] Performing range migration compensation based on the azimuth spectrum in the frequency domain to obtain a compensated frequency domain signal;

[0032] Performing a one-dimensional inverse fast Fourier transform on the compensated frequency domain signal to obtain a compensated range spectrum, and performing a one-dimensional inverse fast Fourier transform on the compensated range spectrum to obtain an imaging result.

[0033] Furthermore, the process of obtaining the phase change of the human body signal includes:

[0034] Read the original data from the reconstructed and corrected SAR image and perform data preprocessing;

[0035] Extract the original phase signal, and perform phase unwrapping whenever the phase difference between consecutive values ​​is greater than or less than ±π to obtain a continuous phase curve;

[0036] A phase difference operation is performed on the unwrapped phase to obtain a phase difference signal.

[0037] Furthermore, the separating of the phase difference signal specifically includes:

[0038] The sliding average filtering algorithm is used to denoise the phase difference signal;

[0039] The denoised signal is band-pass filtered to separate the target person's breathing and heartbeat signals.

[0040] Furthermore, the emotion-related features include time domain analysis, frequency domain analysis, time-frequency analysis, Poincaré diagram, sample entropy and detrended fluctuation analysis, and the current emotion of the target person is obtained based on the emotion-related features through the SVM model.

[0041] The present invention also provides an emotion recognition robot based on millimeter wave radar, comprising:

[0042] The signal transmission and reception module includes a millimeter wave radar, which is used to use the millimeter wave radar to send the modulated millimeter wave signal as a transmission signal to the surrounding environment, receive the reflected echo signal, and obtain an intermediate frequency signal after mixing as the received data;

[0043] Point cloud conversion and static reflection removal module, used to convert the received data into point cloud format, remove static reflections, and obtain dynamic point cloud data and timestamps of the surrounding environment;

[0044] A dynamic point cloud grouping module, configured to group the dynamic point cloud data into clusters;

[0045] The centroid estimation and merging module obtains the centroid estimation value of the target based on the dynamic point cloud data of each cluster;

[0046] A target tracking module is used to track the center of mass estimate to determine the position of the target person at each moment;

[0047] The human body reflection signal sending and receiving module is used to send millimeter wave signals to the target human body through the robot's uniform horizontal movement, and receive the evenly spaced echo signals reflected by the human body;

[0048] A phase error estimation and compensation module, configured to optimize the point spread function of the evenly spaced echo signals to achieve phase compensation;

[0049] SAR image reconstruction module, used to perform SAR image reconstruction correction using a fast frequency domain near-field imaging method to generate a corrected SAR image;

[0050] The micro-vibration displacement change acquisition module is used to extract the phase change of the target distance human body signal from the human body reflection signal obtained by the SAR image reconstruction module to obtain a phase difference signal;

[0051] The heartbeat and breathing signal separation module is used to separate the phase difference signal to obtain the target person's breathing and heartbeat signals;

[0052] A feature extraction module, configured to extract emotion-related features from the breathing and heartbeat signals;

[0053] Model training module, used to train emotion recognition models;

[0054] The emotion recognition module is used to input the emotion-related features obtained in the feature extraction module into the trained emotion recognition model to obtain the current emotion of the target person;

[0055] Motor module, used to control the movement of the robot;

[0056] LCD display module: used to display short-cycle animations corresponding to emotions. When the emotion changes, the animation transition is performed after the current animation cycle is completed, switching to another display mode and providing human-computer interaction functions.

[0057] Communication module: used to receive information from the cloud server and change the emotional state based on the received information;

[0058] The single chip microcomputer module is connected to the communication module and is used to control the motor module and the LCD display module;

[0059] The robot shell module is used to install the millimeter-wave radar board, motor module, single-chip microcomputer module, LCD display module and communication module.

[0060] Compared with existing technologies, this invention significantly improves the convenience, privacy, flexibility, and long-distance accuracy of emotion recognition, and has the following beneficial effects:

[0061] 1. The present invention accurately extracts tiny vibration displacement changes at target distance through the reflection of millimeter wave signals, which can achieve high-precision detection of tiny movements of the target, thereby realizing contactless emotion recognition and protecting user privacy.

[0062] 2. The present invention proposes a solution based on a radar rotation mechanism. By implementing the radar rotation mechanism to track the positioning of human targets, the application flexibility and efficiency of the millimeter-wave radar system in dynamic environments are significantly improved, and the accuracy and real-time performance of target detection and tracking are enhanced.

[0063] 3. This invention innovatively applies synthetic aperture radar technology to the field of emotion recognition. Through PSF optimization and RMA algorithm, it solves the challenge of reduced accuracy in long-distance target recognition of traditional millimeter-wave radar, significantly improving the accuracy of emotion recognition.

[0064] 4. The present invention embeds emotion recognition-related algorithms into hardware and designs intelligent robots, enabling emotion recognition technology to be actually applied in human-computer interaction scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flowchart of emotion recognition in an embodiment of the present invention;

[0066] Figure 2 1 is a phase comparison diagram before and after phase unwrapping in an embodiment of the present invention;

[0067] Figure 3 1 is a time domain waveform diagram of heartbeat and respiration after signal separation in an embodiment of the present invention;

[0068] Figure 4 4 is a diagram of the classification results of the SVM model trained in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention uses millimeter-wave signals to perceive the surrounding environment and obtain radar point cloud data through processing. After eliminating static reflections, the DBSCAN algorithm is used to group dynamic point clouds from the same object into a cluster. Centroid estimation is performed to convert the point clouds from the same object into single-point observations with associated uncertainty. The centroid estimation is tracked using a joint integrated probabilistic data association (JIPDA) tracker to locate the target human body. The human body reflection signal for the millimeter-wave radar signal is then received by horizontally moving uniformly. Phase error is estimated based on point spread function (PSF) optimization for phase compensation. The SAR image is then reconstructed using a fast frequency-domain near-field imaging method (RMA) to generate a corrected SAR image. The processed high-precision signal is used to perceive the tiny vibration displacement caused by the expansion and contraction of the chest due to breathing and the body vibration due to heartbeat. The heartbeat and breathing signals are separated by an IIR bandpass filter. Feature extraction is then performed on the separated signals. Based on this, a support vector machine (SVM) model is trained. The previously calculated features are input into the trained SVM model to obtain the corresponding emotions, which are then displayed, thus realizing contactless and privacy-preserving emotion recognition technology.

[0070] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0071] Example 1

[0072] This embodiment provides an emotion recognition method based on millimeter wave radar, which is applied to indoor scenes. Figure 1 As shown, the following steps are included:

[0073] S1. Transmit millimeter wave signals through millimeter wave radar.

[0074] S2. Mix the transmitted signal and the echo signal to obtain an intermediate frequency signal as the received data. The received data is converted into a point cloud format, and the range rate of all reflections is detected to remove static reflections to ensure accurate detection of dynamic targets.

[0075] S3. At this point, the dynamic point cloud data and timestamps of the surrounding environment are obtained. Based on this, the dynamic point cloud data is grouped into clusters. Specifically, the following steps are included:

[0076] S301. Distance metric: Use Mahalanobis distance as the metric. Mahalanobis distance is defined as:

[0077]

[0078] Among them, x is a sample point, μ is the mean vector of the data set, ∑ is the covariance matrix of the data set, ∑ -1 is the inverse of the covariance matrix;

[0079] S302, parameter setting: setting the parameters of the DBSCAN algorithm, including the epsilon threshold and the minimum number of points per cluster. In this embodiment, the epsilon threshold of the DBSCAN algorithm is 3, and the minimum number of points per cluster is 1;

[0080] S303, clustering: applying the DBSCAN algorithm to group the dynamic point cloud data into clusters.

[0081] S4. After obtaining the point cloud data grouping, the measurement results are merged into the centroid estimate using the standard Gaussian mixture merging algorithm, converting the point cloud from the same object into a single point observation with associated uncertainty. Specifically, the following steps are included:

[0082] S401, each cluster obtained by grouping in step S3 receives clustered point cloud data and its corresponding measurement noise covariance matrix;

[0083] S402. Calculate the centroid position: multiply the measurement value of each point by the inverse of the measurement noise covariance matrix obtained in step S401, and then sum them up:

[0084]

[0085] where z c is the estimated centroid, z i is the measured value at each point, is the inverse of the measurement noise covariance matrix;

[0086] S403, calculate the combined covariance: sum the inverses of all measurement noise covariance matrices to obtain the total inverse covariance matrix, and then take the inverse:

[0087]

[0088] where R c is the noise covariance matrix of the centroids.

[0089] S5. After the centroid is estimated, it can be tracked by a tracker. This embodiment uses a joint integrated probabilistic data association (JIPDA) tracker to track the centroid estimation at each moment, which specifically includes the following steps:

[0090] S501, filter initialization: Use the extended Kalman filter (EKF) to initialize the target state, set low uncertainty and low process noise, initialize the function, and define the initial state and covariance matrix:

[0091]

[0092] Where x0 is the initial state vector, which contains position and velocity, P0 is the initial covariance matrix, which represents the uncertainty of the state estimate, is the uncertainty in position and velocity;

[0093] S502, Data Association: Use the Joint Integrated Probabilistic Data Association (JIPDA) algorithm to associate new observations with existing trajectories and calculate the association probability based on the observations and the measured probabilities of the target states:

[0094]

[0095] Where p(H j |Z) is the hypothesis H given the observed data Z j The probability, p(Z|H j ) is based on the assumption that H j The probability of observing data Z under j ) is the assumption H j The prior probability of is the normalization factor for all hypotheses;

[0096] S503, State Estimation: Use EKF to update the target state estimate, including position and velocity, and update the state covariance matrix based on measurement noise and process noise:

[0097] x k|k =x k|k-1 +K k (z k -Hx k|k-1 )

[0098] where x k|k is the updated state estimate, x k|k-1 is the predicted state estimate, K k is the Kalman gain matrix, z k is the current observation value, H is the observation matrix that maps the state space to the observation space;

[0099] S504, trajectory management: When the existence probability p(Hj The trajectory is confirmed when the probability of the target's existence exceeds the set threshold of 95%, and the trajectory is deleted when the probability of the target's existence falls below the set threshold of 0.0001%.

[0100] At step 6, the target person's position has been tracked. However, because the target person is too far away, the directly acquired signal will be distorted. Therefore, Synthetic Aperture Radar (SAR) is needed to improve accuracy. First, the millimeter-wave radar is moved horizontally at a constant speed to send millimeter-wave signals to the target person. The evenly spaced echo signals reflected by the person are received, allowing for efficient imaging.

[0101] S7. After acquiring the signal, phase compensation can be performed by optimizing the point spread function (PSF). The process of estimating the phase error based on PSF optimization and performing phase compensation includes the following steps:

[0102] S701. First, due to the correlation of phase errors between different targets, a point target is manually designated as a reference in the imaging scene to ensure that both the reference target and the imaging target are observed by the radar. The round-trip propagation error between the two targets is expressed as:

[0103]

[0104] ΔR=|(ΔR1|-|ΔR2|)

[0105] The ideal antenna position for a single virtual antenna is (x′ i ,z i ′), and the actual antenna position includes the motion error Δz in the z-axis i , so the actual position is (x′ i ,z i ′+Δz i ), where ΔR1 and ΔR2 represent the round-trip propagation errors of the target (x1, z1) and the target (x2, z2), respectively, and ΔR represents the difference between ΔR1 and ΔR2.

[0106] Obviously, the smaller ΔR is, the closer the phase errors of the two targets are. According to the distribution of ΔR, a point target is set, and its estimated phase error can be very close to the phase error of the imaging target;

[0107] S702. Then, the optimized PSF is obtained by fitting the phase history of the point target to the corresponding ideal quadratic phase history. Through this process, the phase error of the point target can be measured by quantifying the difference between the actual phase history of the point target and the ideal phase history. Utilizing the principle of frequency modulated continuous wave (FMCW), the two targets in the range domain are separated by performing an FFT along each reflected pulse. Since the phase history of the ideal trajectory point target follows a quadratic curve, the ideal phase change of the reference point target can be estimated by optimizing the following function:

[0108]

[0109] in, represents the estimated ideal phase change of the received signal of the reference point target at the i-th position, φ i is the corresponding perturbation phase change extracted by FFT, N is the number of received pulses, α, β, and γ are the coefficients of the target curve, and the function finds the optimal α, β, and γ values ​​that minimize the difference between the actual phase change of the reference object and the ideal phase change.

[0110] Once the ideal phase history is estimated, the phase error caused by hand-held motion can be compensated as:

[0111]

[0112] in represents the motion error to be compensated, s i Represents motion errors that are corrupted by the received signal.

[0113] At this point, the phase in the signal has been corrected, which can better reflect the true reflection characteristics of the target and improve the imaging quality.

[0114] S8, at this time, the signal after PSF optimization is obtained, that is, the motion error after phase error compensation Phase compensation of radar signals is performed using the fast frequency domain near-field imaging method (RMA) to reconstruct the SAR image. The process of generating the corrected SAR image specifically includes the following steps:

[0115] S801, Range Spectrum Calculation: Perform a one-dimensional fast Fourier transform on each pulse echo to obtain the range spectrum, analyze the reflection characteristics of the target at different distances, and improve the accuracy of target detection and positioning:

[0116]

[0117] Where t represents time, x represents antenna position, k r represents the distance frequency, S(k r ,x) represents the distance spectrum, s(t,x) represents the time domain signal, represents one-dimensional fast Fourier transform;

[0118] S802, azimuth spectrum calculation: Perform a one-dimensional fast Fourier transform on each distance unit to obtain an azimuth spectrum, thereby improving the spatial resolution of the imaging system.

[0119]

[0120] Where S(k r ,kx ) is the azimuthal spectrum, k x is the azimuth frequency;

[0121] S803, Range Migration Compensation: Perform range migration compensation (RMC) in the frequency domain to correct the range offset caused by the target position change:

[0122]

[0123] Where S′(k r ,k x ) is the compensated frequency domain signal;

[0124] S804, azimuth inverse Fourier transform: Perform a one-dimensional inverse fast Fourier transform on the compensated frequency domain signal to obtain the compensated range spectrum, maintaining the integrity and accuracy of the signal:

[0125]

[0126] Where S′(k r ,x) represents the distance spectrum after compensation;

[0127] S805, range inverse Fourier transform: Perform one-dimensional inverse fast Fourier transform on the compensated range spectrum to obtain the imaging result:

[0128]

[0129] Where s′(t,x) represents the time domain signal after compensation.

[0130] S9. After obtaining the high-precision target human body signal, perform signal processing calculation to obtain the phase change of the human body signal. The specific steps are as follows:

[0131] S901. Data preprocessing: For the raw data read, some data preprocessing operations need to be performed, including data format conversion, correction, and data reorganization, so as to make the data clearer and more accurate for subsequent analysis.

[0132] S902, Range-Dimensional FFT: Perform an FFT on a single beat signal to obtain its spectrum. The calculated spectrum peak corresponds to the distance to the target, which is called the range-dimensional FFT. The range-dimensional FFT transform improves distance measurement accuracy and helps suppress noise and interference.

[0133] The specific formula for the relationship between the maximum detection range of FMCW radar and the sampling rate of ADC is:

[0134]

[0135] where d maxIndicates the maximum detection distance of the radar, F s represents the sampling rate, c represents the speed of light, and S represents the slope of the linear frequency modulation pulse in the frequency domain.

[0136] The spectrum peak, i.e., the distance to the target being measured, can be obtained from the specific formula for the relationship between the maximum detection range of the FMCW radar and the sampling rate of the ADC.

[0137] S903. Extracting the original phase signal: The data processed in steps S901 and S902 is first filtered through static clutter to remove some DC components, eliminate reflections from static objects that do not change over time, and amplify reflections from the human body. Phase changes correspond to the combined displacement caused by chest expansion and contraction due to breathing and body vibrations due to heartbeats. It then examines the RF wave phase in relation to the distance traveled, using the formula:

[0138]

[0139] Where Φ(t) represents the phase of the signal, λ represents the wavelength, d(t) represents the travel distance, and t represents the time variable.

[0140] S904, Phase Unwrapping: Phase unwrapping is also called phase unwinding. Since the phase value is between [-π, π], phase unwrapping is required to obtain the actual displacement curve. Therefore, whenever the phase difference between consecutive values ​​is greater than or less than ±π, the phase unwrapping is obtained by subtracting 2π from the phase, such as Figure 2 As shown in FIG, by phase unwrapping, the 2π modular operation effect can be eliminated from the original phase signal to obtain a continuous phase curve.

[0141] S905, Phase Difference: By continuously subtracting the phase of the current sampling point from the previous sampling point, and subtracting consecutive phase values, a phase difference operation is performed on the expanded phase. This helps to enhance the heartbeat signal and eliminate the phase drift existing in the hardware receiver, and suppress the respiratory signal and its harmonics.

[0142] S10. At this point, the phase difference signal after the above processing can be separated. However, the influence of environmental noise on the signal needs to be removed. Therefore, the phase change obtained in step S9 is subjected to sliding average filtering to remove noise. Then, the respiration and heartbeat signals are separated by an IIR bandpass filter. Finally, the heart rate and respiration rate are calculated by the frequency domain FFT algorithm. The specific steps include:

[0143] S1001, sliding average filtering: before separating the phase difference signal obtained in step S9, a sliding average filtering algorithm is used to remove the influence of impulse noise caused by the test environment on the signal;

[0144] S1002. Bandpass filter the output respiratory signal and perform FFT spectrum analysis: Use an IIR bandpass filter to process the phase signal obtained above, perform a fast Fourier transform on the respiratory signal separated by the bandpass filter to obtain the respiratory frequency, and then convert it into the respiratory rate. The specific formula is:

[0145] RR=f RR ×60(times / min)

[0146] where f RR represents respiratory frequency, and RR represents respiratory rate;

[0147] S1003. Band-pass filter outputs the heartbeat signal and performs FFT spectrum analysis: Perform a fast Fourier transform on the heartbeat signal separated by the band-pass filter to obtain the heartbeat frequency, which is then converted into the heart rate. The specific formula is:

[0148] HR = f HR ×60(times / min)

[0149] where f HR represents respiratory rate, and HR represents respiratory rate.

[0150] like Figure 3 As shown, Figure 3 (a) is the processed respiratory time domain waveform, and (b) is the processed heartbeat time domain waveform. Both are within the range of normal human breathing and heartbeat, indicating that the processing method is effective.

[0151] S11. At this point, the heartbeat and breathing signals of the target person have been obtained. Next, it is necessary to extract emotion-related features from the signals and then analyze the emotions. Therefore, feature extraction is performed on the heartbeat and breathing signals obtained from step S10. These features can be divided into time domain analysis, frequency domain analysis, time-frequency analysis, Poincaré diagram, sample entropy and detrended fluctuation analysis, thereby ignoring information irrelevant to the target, reducing the amount of calculation, and improving the generalization ability of the model.

[0152] S12. Currently, only relevant features related to emotions can be obtained, and the corresponding emotions of the target person cannot be obtained. Therefore, in order to obtain the emotions of the target person, this embodiment uses the SVM model to perform emotion recognition.

[0153] In this embodiment, the SVM model uses relevant data from the MAHNOB-HCI-Tagging Database public database. The process of training the model includes the following steps:

[0154] S1201, extracting training data: using relevant data from the MAHNOB-HCI-Tagging Database public database to calculate corresponding features in the heartbeat and respiratory signals, and extracting the corresponding valence arousal index and emotion classification;

[0155] S1202, dividing the training set and the test set: 80% of the data is divided for training, and 20% of the data is reserved for testing the training results;

[0156] S1203, defining an SVM model: using the training data divided in step S1202, training an SVM model for valence and arousal, wherein the model normalizes the feature data during training, specifies the kernel function as a radial basis function, and automatically selects a scale parameter of the kernel function;

[0157] S1204, training an SVM model: Based on the Error-Correcting Output Codes (ECOC) method, a multi-class classification model is trained, using the previously defined SVM template as the base classifier, and inputting the features and labels of the training set for model training;

[0158] S1205 , observing the training effect: using the test data divided in step S1202 to test the SVM model trained in step S1203 , and calculating the prediction accuracy of the valence and arousal index.

[0159] The classification results of the SVM model in this embodiment are as follows Figure 4 As shown, the classification accuracy reaches 74.165%.

[0160] S13: Input the relevant features calculated in step S11 into the trained SVM model to obtain the current emotion of the target person and complete emotion recognition. The specific steps include:

[0161] S1301 : Input the relevant features calculated in step S11 into the trained SVM model to obtain corresponding valence and arousal index.

[0162] S1302: Using the SVM model, convert the valence and arousal index obtained in step S1301 into emotions to obtain the current emotion of the target person.

[0163] S14: Display the emotion identified in step S13. In this embodiment, the current emotion and the corresponding emotion suggestion are displayed in emoticons. For example, if the current emotion is anger, the user is advised to take a deep breath and look around to calm down.

[0164] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0165] Example 2

[0166] This embodiment provides an emotion recognition robot based on millimeter-wave radar, including a signal transmission and reception module, a point cloud conversion and static reflection elimination module, a dynamic point cloud grouping module, a centroid estimation and merging module, a target tracking module, a human body reflection signal transmission and reception module, a phase error estimation and compensation module, a SAR image reconstruction module, a small vibration displacement change acquisition module, a heartbeat and breathing signal separation module, a feature extraction module, a model training module, an emotion recognition module, a motor module, an LCD display module, an ESP8266 communication module, a single-chip microcomputer module, a robot shell module, and a computer program signal that can be run on the processor. Among them, the transmitting and receiving module includes a millimeter wave radar, which is used to use the millimeter wave radar to send the modulated millimeter wave signal as a transmitting signal to the surrounding environment, and receive the reflected echo signal, and obtain the intermediate frequency signal after mixing as the received data; the point cloud conversion and static reflection elimination module is used to convert the received data into a point cloud format, remove static reflections, and obtain dynamic point cloud data and timestamps of the surrounding environment; the dynamic point cloud grouping module is used to group the dynamic point cloud data into clusters; the centroid estimation and merging module obtains the centroid estimate of the target based on the dynamic point cloud data of each cluster, and converts the point cloud from the same object into a single point observation with related uncertainty; the target tracking module is used to track the centroid estimate and determine the target person's position at each moment; the human body reflection signal sending and receiving module is used to send the millimeter wave signal to the target human body through the robot's uniform horizontal movement, and receive the evenly spaced echo signals reflected by the human body. ; The phase error estimation and compensation module is used to optimize the point spread function of the evenly spaced echo signal to achieve phase compensation; the SAR image reconstruction module is used to use the fast frequency domain near-field imaging method to perform SAR image reconstruction correction to generate a corrected SAR image; the small vibration displacement change acquisition module is used to extract the phase change of the target distance human body signal from the human body reflection signal obtained by the SAR image reconstruction module, obtain the small vibration displacement change, and then obtain the phase difference signal; the heartbeat and breathing signal separation module is used to process the phase change obtained in the small vibration displacement change acquisition module through an IIR bandpass filter to separate the breathing and heartbeat signals of the target person; the feature extraction module is used to extract emotion-related features from the breathing and heartbeat signals; the model training module is used to train the emotion recognition model. In this embodiment, the SVM model is used, and the data in the MAHNOB-HCI-Tagging Database public database is used to train the model; the emotion recognition module is used to input the emotion-related features obtained in the feature extraction module into the trained emotion recognition model to obtain the current emotion of the target person; the motor module is used to control the movement of the robot to facilitate the positioning of the human target and improve the accuracy by using the RMA algorithm through PSF optimization;The LCD display module is used to display short-cycle animations corresponding to emotions. When the emotion changes, the animation transitions after the current animation cycle, switching to another emoji display. The TFT touch screen adds more active human-computer interaction functions. The communication module is used to receive information from the cloud server and change the emotional state based on the received information. In this embodiment, the ESP8266 communication module is used, which can receive information from conventional Wi-Fi and hotspot devices. The single-chip microcomputer module is connected to the communication module and is responsible for LCD screen control, motor control, user perception and communication task control. The user perception and motor control clocks maintain a low priority and always run to adjust the posture. When the emotional state is received, the LCD screen emoji animation is controlled. The robot shell module is used to install the millimeter wave radar board, motor module, single-chip microcomputer module, LCD display module and communication module, so as to achieve a compact and optimized internal structure and a refined and beautiful appearance of the robot.

[0167] It should be noted that each module in the above system corresponds to the specific steps of the method provided in Example 1 of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in this embodiment, please refer to the method provided in Example 1 of the present invention.

[0168] In this example, the test scenario involved a user indoors, at a distance from the emotion recognition robot, not facing it directly, with their chest level with the robot. The millimeter wave signal was transmitted from a TI IWR1443 millimeter wave radar, a 76-81 GHz FMCW millimeter wave signal. The target person watched a video clip that triggered a specific emotional state.

[0169] Specifically, this example involved three volunteers as target subjects. Their chests were positioned on the same horizontal plane as the emotion recognition robot, 5 meters apart and not directly facing it. During the test, they were allowed to stand, sit, or walk while measuring different emotions multiple times. The final detection accuracy was approximately 72%.

[0170] In another embodiment, the emotion recognition robot can be connected to a cloud server. The radar emission and echo signals collected by the emotion recognition robot are uploaded to the cloud server. The cloud server processes the signals according to the method provided in Example 1 and then displays the recognized emotions on the robot's display screen. In this embodiment, the cloud server is an Alibaba Cloud server with an ECS-ARM architecture, a Yitian c8y processor, and 2 cores and 4GB of RAM.

[0171] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for emotion recognition based on millimeter wave radar, characterized in that: The following steps are involved: Receive the millimeter-wave radar's transmitted signal and the corresponding echo signal, mix them to obtain the intermediate frequency signal as the received data, convert the received data into a point cloud format, remove static reflections, and obtain dynamic point cloud data and timestamps of the surrounding environment; Grouping the dynamic point cloud data into clusters; Obtaining a target's center of mass estimate based on each cluster of dynamic point cloud data, tracking the center of mass estimate, and determining the target person's position at each moment; Send a millimeter wave signal moving horizontally at a uniform speed to the target person, and receive the evenly spaced echo signals reflected by the target person; performing point spread function optimization on the evenly spaced echo signals, and performing SAR image reconstruction and correction using a fast frequency domain near-field imaging method; The phase change of the human body signal is calculated based on the reconstructed and corrected SAR image to obtain the phase difference signal, which is then separated to obtain the breathing and heartbeat signals of the target person. Emotion-related features are extracted from the breathing and heartbeat signals, and the current emotion of the target person is identified based on the emotion-related features.

2. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: Grouping the dynamic point cloud data into clusters specifically includes: Using Mahalanobis distance as the metric, DBSCAN algorithm is applied to cluster the dynamic point cloud data into clusters.

3. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The process of obtaining the estimated centroid value includes: Calculate the measurement noise covariance matrix corresponding to each cluster of point cloud data; The centroid position is calculated based on the sum of the products of the inverse of the measurement noise covariance matrix and the measurement value of each point in each cluster point cloud data, and the noise covariance matrix of the centroid is obtained according to the inverse of the sum of the inverses of all measurement noise covariance matrices.

4. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: Tracking the centroid estimate using a joint integrated probabilistic data association algorithm includes the following steps: Initialize the target state using the extended Kalman filter; Use the joint integrated probabilistic data association algorithm to calculate the association probability and associate the new observation data with the existing trajectory; Update the target's state estimate using an extended Kalman filter; Determine whether to confirm or delete the track based on the probability of the target's existence.

5. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The point spread function optimization specifically includes: A point target is designated as a reference target in the imaging scene, and the round-trip propagation error between the reference target and the imaging target is calculated; Based on the difference distribution of the round-trip propagation error between different point targets, a final point target is set, and phase error compensation is obtained based on the final point target phase history and the corresponding ideal quadratic phase history to obtain a signal after point spread function optimization.

6. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The SAR image reconstruction correction specifically includes: Performing a one-dimensional fast Fourier transform on each pulse echo in the signal after point spread function optimization to obtain a range spectrum, and performing a one-dimensional fast Fourier transform on each range spectrum to obtain an azimuth spectrum; Performing range migration compensation based on the azimuth spectrum in the frequency domain to obtain a compensated frequency domain signal; Performing a one-dimensional inverse fast Fourier transform on the compensated frequency domain signal to obtain a compensated range spectrum, and performing a one-dimensional inverse fast Fourier transform on the compensated range spectrum to obtain an imaging result.

7. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The process of obtaining the phase change of the human body signal includes: Read the original data from the reconstructed and corrected SAR image and perform data preprocessing; Extract the original phase signal, and perform phase unwrapping whenever the phase difference between consecutive values ​​is greater than or less than ±π to obtain a continuous phase curve; A phase difference operation is performed on the unwrapped phase to obtain a phase difference signal.

8. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The separation of the phase differential signal specifically includes: The sliding average filtering algorithm is used to denoise the phase difference signal; The denoised signal is band-pass filtered to separate the target person's breathing and heartbeat signals.

9. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The emotion-related features include time domain analysis, frequency domain analysis, time-frequency analysis, Poincaré diagram, sample entropy and detrended fluctuation analysis, and the current emotion of the target person is obtained based on the emotion-related features through the SVM model.

10. An emotion recognition robot based on millimeter wave radar, characterized in that: include: The signal transmission and reception module includes a millimeter wave radar, which is used to use the millimeter wave radar to send the modulated millimeter wave signal as a transmission signal to the surrounding environment, receive the reflected echo signal, and obtain an intermediate frequency signal after mixing as the received data; Point cloud conversion and static reflection removal module, used to convert the received data into point cloud format, remove static reflections, and obtain dynamic point cloud data and timestamps of the surrounding environment; A dynamic point cloud grouping module, configured to group the dynamic point cloud data into clusters; The centroid estimation and merging module obtains the centroid estimation value of the target based on the dynamic point cloud data of each cluster; A target tracking module is used to track the center of mass estimate to determine the position of the target person at each moment; The human body reflection signal sending and receiving module is used to send millimeter wave signals to the target human body through the robot's uniform horizontal movement, and receive the evenly spaced echo signals reflected by the human body; A phase error estimation and compensation module, configured to optimize the point spread function of the evenly spaced echo signals to achieve phase compensation; SAR image reconstruction module, used to perform SAR image reconstruction correction using a fast frequency domain near-field imaging method to generate a corrected SAR image; The micro-vibration displacement change acquisition module is used to extract the phase change of the target distance human body signal from the human body reflection signal obtained by the SAR image reconstruction module to obtain a phase difference signal; The heartbeat and breathing signal separation module is used to separate the phase difference signal to obtain the target person's breathing and heartbeat signals; A feature extraction module, configured to extract emotion-related features from the breathing and heartbeat signals; Model training module, used to train emotion recognition models; The emotion recognition module is used to input the emotion-related features obtained in the feature extraction module into the trained emotion recognition model to obtain the current emotion of the target person; Motor module, used to control the movement of the robot; LCD display module: used to display short-cycle animations corresponding to emotions. When the emotion changes, the animation transition is performed after the current animation cycle is completed, switching to another display mode and providing human-computer interaction functions. Communication module: used to receive information from the cloud server and change the emotional state based on the received information; The single chip microcomputer module is connected to the communication module and is used to control the motor module and the LCD display module; The robot shell module is used to install the millimeter-wave radar board, motor module, single-chip microcomputer module, LCD display module and communication module.

Images