Emotion recognition method based on millimeter wave radar and robot

Abstract

The invention relates to an emotion recognition method based on a millimeter-wave radar and a robot. The method comprises the following steps: receiving a transmitting signal of the millimeter-wave radar and a corresponding echo signal, converting the transmitting signal and the corresponding echo signal into a point cloud format, and removing static reflection to obtain dynamic point cloud data and a timestamp of a surrounding environment; grouping the dynamic point cloud data into clusters; obtaining a centroid estimation value of the target, and tracking the centroid estimation value; sending the millimeter wave signal which horizontally moves at a constant speed to a target person, and receiving echo signals which are reflected by the target person and are uniformly spaced; point spread function optimization is carried out, and SAR image reconstruction is carried out by using a fast frequency domain near-field imaging method; calculating to obtain phase change of the human body signal, and separating to obtain respiration and heartbeat signals of the target person; and extracting emotion related features to obtain the current emotion of the target person. Compared with the prior art, the specific position of the human body can be accurately recognized, and non-contact privacy protection emotion recognition can be remotely achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to an emotion recognition method and a robot based on millimeter wave radar. Background Art

[0002] With the advancement of technology, smart homes and the smart Internet of Things behind them are gradually showing their irreplaceable advantages in the development of the elderly care industry. However, on the basis of basic living needs, the spiritual needs of the elderly group also need to be given enough care and attention.

[0003] At present, emotional care for the elderly is still at a stage that relies heavily on manpower. Although there are already some smart IoT devices that can provide assistance by detecting basic physiological indicators of the human body, such as a wearable physiological indicator detection device disclosed in Chinese utility model patent CN219742686U, it is still imperfect. On the one hand, the detection of physiological indicators often requires the target to wear specific sensors, and often requires the target to be in a specific position or even maintain a specific posture, which can easily cause physical discomfort to the elderly and hinder all-weather continuous detection; on the other hand, the analysis and interpretation of sensor data to the generation of feedback currently still basically relies on manpower, and it is impossible to provide the elderly with appropriate feedback and emotional care in real time, and the reliance on manpower also hinders all-weather continuous care. Summary of the invention

[0004] One of the purposes of the present invention is to provide an emotion recognition method and robot based on millimeter-wave radar. Based on the non-contact and high-resolution characteristics of millimeter-wave radar, a signal processing method is designed, and the corresponding emotions are obtained through machine learning, thereby overcoming the problem of privacy exposure when using other technologies to identify emotions and realizing contactless emotion recognition with privacy protection.

[0005] The second purpose of the present invention is to provide an emotion recognition method and robot based on millimeter-wave radar, which solves the problem that the user can only maintain a relatively fixed posture during emotion recognition in the original technology through rotational movement signal analysis based on millimeter-wave radar, thereby being able to track the target position of the human body and achieve more flexible emotion recognition.

[0006] The third purpose of the present invention is to provide an emotion recognition method and robot based on millimeter-wave radar, and long-distance high-precision target recognition based on multi-aperture radar. The problem of reduced accuracy of emotion recognition due to decreased accuracy of millimeter-wave radar at long distances is solved through point spread function (PSF) optimization and fast frequency domain near-field imaging method (RMA) algorithm, thereby obtaining a solution for long-distance and high-precision recognition of target emotions.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] An emotion recognition method based on millimeter wave radar comprises the following steps:

[0009] Receive the transmission signal and the corresponding echo signal of the millimeter-wave radar, mix them to obtain the intermediate frequency signal as the received data, convert the received data into point cloud format, remove the static reflection, and obtain the dynamic point cloud data and timestamp of the surrounding environment;

[0010] Grouping the dynamic point cloud data into clusters;

[0011] Obtaining a centroid estimate of the target based on each cluster of dynamic point cloud data, tracking the centroid estimate, and determining the position of the target person at each moment;

[0012] Send a millimeter wave signal moving horizontally at a uniform speed to a target person, and receive evenly spaced echo signals reflected by the target person;

[0013] Performing point spread function optimization on the evenly spaced echo signals, and performing SAR image reconstruction correction using a fast frequency domain near-field imaging method;

[0014] The phase change of the human body signal is calculated based on the reconstructed and corrected SAR image to obtain the phase difference signal, and the phase difference signal is separated to obtain the breathing and heartbeat signals of the target person;

[0015] Emotion-related features are extracted from the breathing and heartbeat signals, and the current emotion of the target person is obtained based on the emotion-related features.

[0016] Furthermore, grouping the dynamic point cloud data into clusters specifically includes:

[0017] Using Mahalanobis distance as the metric, DBSCAN algorithm is applied to group the dynamic point cloud data into clusters.

[0018] Furthermore, the process of obtaining the estimated value of the centroid includes:

[0019] Calculate the measurement noise covariance matrix corresponding to each cluster of point cloud data;

[0020] The centroid position is calculated based on the sum of the products of the inverse of the measurement noise covariance matrix and the measurement value of each point in each cluster point cloud data, and the noise covariance matrix of the centroid is obtained according to the inverse of the sum of the inverses of all measurement noise covariance matrices.

[0021] Further, tracking the centroid estimate using a joint integrated probabilistic data association algorithm comprises the following steps:

[0022] Initialize the target state using the extended Kalman filter;

[0023] Use the joint integrated probabilistic data association algorithm to calculate the association probability and associate the new observation data with the existing trajectories;

[0024] Update the target's state estimate using an extended Kalman filter;

[0025] Determine whether to confirm or delete the track based on the probability of the target's existence.

[0026] Furthermore, the point spread function optimization specifically includes:

[0027] A point target is designated as a reference target in the imaging scene, and the round-trip propagation error between the reference target and the imaging target is calculated;

[0028] Based on the difference distribution of the round-trip propagation error between different point targets, a final point target is set, and phase error compensation is obtained based on the final point target phase history and the corresponding ideal quadratic phase history to obtain a signal after point spread function optimization.

[0029] Furthermore, the SAR image reconstruction correction specifically includes:

[0030] Performing one-dimensional fast Fourier transform on each pulse echo in the signal after point spread function optimization to obtain a range spectrum, and performing one-dimensional fast Fourier transform on each range spectrum to obtain an azimuth spectrum;

[0031] Performing distance migration compensation based on the azimuth spectrum in the frequency domain to obtain a compensated frequency domain signal;

[0032] The compensated frequency domain signal is subjected to one-dimensional inverse fast Fourier transform to obtain a compensated range spectrum, and the compensated range spectrum is subjected to one-dimensional inverse fast Fourier transform to obtain an imaging result.

[0033] Furthermore, the process of obtaining the phase change of the human body signal includes:

[0034] Read the raw data from the reconstructed and corrected SAR image and perform data preprocessing;

[0035] Extract the original phase signal, and perform phase unwrapping whenever the phase difference between consecutive values ​​is greater than or less than ±π to obtain a continuous phase curve;

[0036] A phase difference operation is performed on the unwrapped phase to obtain a phase difference signal.

[0037] Furthermore, the separation of the phase difference signal specifically includes:

[0038] The sliding average filtering algorithm is used to denoise the phase difference signal;

[0039] The denoised signal is bandpass filtered to separate the breathing and heartbeat signals of the target person.

[0040] Furthermore, the emotion-related features include time domain analysis, frequency domain analysis, time-frequency analysis, Poincaré diagram, sample entropy and detrended fluctuation analysis, and the current emotion of the target person is obtained based on the emotion-related feature recognition through the SVM model.

[0041] The present invention also provides an emotion recognition robot based on millimeter wave radar, comprising:

[0042] The signal transmitting and receiving module includes a millimeter wave radar, which is used to use the millimeter wave radar to send the modulated millimeter wave signal as a transmitting signal to the surrounding environment, and receive the reflected echo signal, and obtain an intermediate frequency signal after mixing as receiving data;

[0043] Point cloud conversion and static reflection elimination module, used to convert the received data into point cloud format, remove static reflections, and obtain dynamic point cloud data and timestamp of the surrounding environment;

[0044] A dynamic point cloud grouping module, used for grouping the dynamic point cloud data into clusters;

[0045] The centroid estimation and merging module obtains the centroid estimation value of the target based on the dynamic point cloud data of each cluster;

[0046] A target tracking module, used to track the center of mass estimation value to determine the position of the target person at each moment;

[0047] The human body reflection signal sending and receiving module is used to send millimeter wave signals to the target human body through the robot's uniform horizontal movement, and receive the evenly spaced echo signals reflected by the human body;

[0048] A phase error estimation and compensation module, used for optimizing the point spread function of the evenly spaced echo signals to achieve phase compensation;

[0049] A SAR image reconstruction module is used to perform SAR image reconstruction correction using a fast frequency domain near-field imaging method to generate a corrected SAR image;

[0050] A small vibration displacement change acquisition module is used to extract the phase change of the target distance human body signal from the human body reflection signal obtained by the SAR image reconstruction module to obtain a phase difference signal;

[0051] The heartbeat and breathing signal separation module is used to separate the phase difference signal to obtain the breathing and heartbeat signals of the target person;

[0052] A feature extraction module, used to extract emotion-related features from the breathing and heartbeat signals;

[0053] Model training module, used to train emotion recognition model;

[0054] The emotion recognition module is used to input the emotion-related features obtained in the feature extraction module into the trained emotion recognition model to obtain the current emotion of the target person;

[0055] Motor module, used to control the movement of the robot;

[0056] LCD display module: used to display short-cycle animations corresponding to emotions. When the emotion changes, the animation transition is performed after the current animation cycle is completed, switching to another display mode, and providing human-computer interaction functions;

[0057] Communication module: used to receive information from the cloud server and change the emotional state based on the received information;

[0058] The single chip microcomputer module is connected with the communication module and is used to realize the control of the motor module and the LCD display module;

[0059] The robot shell module is used to install the millimeter wave radar board, motor module, single-chip computer module, LCD display module and communication module.

[0060] Compared with the prior art, the present invention has significantly improved the convenience, privacy, flexibility, and long-distance accuracy of emotion recognition, and has the following beneficial effects:

[0061] 1. The present invention accurately extracts tiny vibration displacement changes at target distances through the reflection of millimeter wave signals, and can achieve high-precision detection of tiny movements of the target, thereby realizing contactless emotion recognition and protecting user privacy.

[0062] 2. The present invention proposes a solution based on a radar rotation mechanism. By implementing the radar rotation mechanism to track the positioning of human targets, the application flexibility and efficiency of the millimeter-wave radar system in a dynamic environment are significantly improved, and the accuracy and real-time performance of target detection and tracking are enhanced.

[0063] 3. The present invention innovatively applies synthetic aperture radar technology to the field of emotion recognition. Through PSF optimization and RMA algorithm, it solves the challenge of reduced accuracy in long-distance target recognition of traditional millimeter-wave radar, and significantly improves the accuracy of emotion recognition.

[0064] 4. The present invention embeds emotion recognition related algorithms into hardware and designs intelligent robots, so that emotion recognition technology can be actually applied in human-computer interaction scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of emotion recognition in an embodiment of the present invention;

[0066] Figure 2 is a phase comparison diagram before and after phase unwrapping in an embodiment of the present invention;

[0067] Figure 3 is a time domain waveform diagram of heartbeat and respiration after signal separation in an embodiment of the present invention;

[0068] Figure 4 It is a classification result diagram of the training SVM model in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention senses the surrounding environment through millimeter wave signals, obtains radar point cloud data through processing, and uses the DBSCAN algorithm to group dynamic point clouds from the same object into a cluster after eliminating static reflections, performs centroid estimation to convert point clouds from the same object into single point observations with associated uncertainties, and uses a joint integrated probabilistic data association (JIPDA) tracker to track the centroid estimation, locate the target human body position, and then uniformly move horizontally to receive the human body reflection signal for the millimeter wave radar signal, and use the point spread function (PSF)-based optimization to estimate the phase error for phase compensation, and then use the fast frequency domain near-field imaging method (RMA) to reconstruct the SAR image to generate a corrected SAR image, and use the processed high-precision signal to sense the small vibration displacement caused by the expansion and contraction of the chest due to breathing and the body vibration due to the heartbeat, and separate the heartbeat and breathing signals through an IIR bandpass filter, and then perform feature extraction on the separated signals, and on this basis, train an SVM model, input the previously calculated features into the trained SVM model, obtain the corresponding emotions, and then display them, thereby realizing a contactless privacy-protecting emotion recognition technology.

[0070] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0071] Example 1

[0072] This embodiment provides an emotion recognition method based on millimeter wave radar, which is applied to indoor scenes. Figure 1 As shown, the following steps are included:

[0073] S1. Transmit millimeter wave signals through millimeter wave radar.

[0074] S2. Mix the transmitted signal and the echo signal to obtain an intermediate frequency signal as the received data. The received data is converted into a point cloud format, and the static reflections are removed by detecting the range rate of all reflections to ensure accurate detection of dynamic targets.

[0075] S3. At this time, the dynamic point cloud data and timestamp of the surrounding environment are obtained, and on this basis, the dynamic point cloud data is grouped into clusters, which specifically includes the following steps:

[0076] S301, distance metric: Use Mahalanobis distance as the metric, Mahalanobis distance is defined as;

[0077]

[0078] Among them, x is a sample point, μ is the mean vector of the data set, ∑ is the covariance matrix of the data set, ∑ -1 is the inverse of the covariance matrix;

[0079] S302, parameter setting: setting the parameters of the DBSCAN algorithm, including the epsilon threshold and the minimum number of points for each cluster. In this embodiment, the epsilon threshold of the DBSCAN algorithm is 3, and the minimum number of points for each cluster is 1;

[0080] S303, clustering: applying the DBSCAN algorithm to group the dynamic point cloud data into clusters.

[0081] S4. After obtaining the point cloud data grouping, the measurement results are merged into the centroid estimate using the standard Gaussian mixture merging algorithm, and the point cloud from the same object is converted into a single point observation with associated uncertainty. Specifically, it includes the following steps:

[0082] S401, each cluster obtained by grouping in step S3 receives clustered point cloud data and its corresponding measurement noise covariance matrix;

[0083] S402, calculate the centroid position: multiply the measurement value of each point by the inverse of the measurement noise covariance matrix obtained in step S401, and then sum them up:

[0084]

[0085] where z c is the estimated centroid, z i is the measured value at each point, is the inverse of the measurement noise covariance matrix;

[0086] S403, calculate the combined covariance: sum the inverses of all measurement noise covariance matrices to obtain a total inverse covariance matrix, and then take the inverse:

[0087]

[0088] Where R c is the noise covariance matrix of the centroids.

[0089] S5. After the centroid is estimated, it can be tracked by a tracker. This embodiment uses a joint integrated probabilistic data association (JIPDA) tracker to track the centroid estimation at each moment, which specifically includes the following steps:

[0090] S501, filter initialization: Use the extended Kalman filter (EKF) to initialize the target state, set low uncertainty and low process noise, initialize the function, and define the initial state and covariance matrix:

[0091]

[0092] Where x0 is the initial state vector, containing position and velocity, P0 is the initial covariance matrix, representing the uncertainty of the state estimate, is the uncertainty in position and velocity;

[0093] S502, data association: Use the Joint Integrated Probabilistic Data Association (JIPDA) algorithm to associate new observation data with existing trajectories, and calculate the association probability based on the observation data and the measurement probability of the target state:

[0094]

[0095] Where p(H j |Z) is the hypothesis H given the observed data Z j The probability, p(Z|H j ) is based on the assumption that H j The probability of observing data Z under j ) is the assumption that H j The prior probability of is the normalization factor for all hypotheses;

[0096] S503, state estimation: Use EKF to update the state estimation of the target, including position and velocity, and update the state covariance matrix based on measurement noise and process noise:

[0097] x k|k =x k|k-1 +K k (z k -Hx k|k-1 )

[0098] where x k|k is the updated state estimate, x k|k-1 is the predicted state estimate, K k is the Kalman gain matrix, z k is the current observation value, H is the observation matrix that maps the state space to the observation space;

[0099] S504, track management: When the existence probability p(Hj When the probability of the target's existence is lower than the set threshold of 0.0001%, the trajectory is deleted.

[0100] S6. At this time, the target human position has been tracked, but because the target human is too far away, the signal directly collected will be distorted, so it is necessary to improve the accuracy through synthetic aperture radar (SAR). First, the millimeter wave signal is sent to the target human body by moving the millimeter wave radar horizontally at a uniform speed, and the evenly spaced echo signals reflected by the human body are received to apply an efficient imaging method.

[0101] S7. After the signal is collected, phase compensation can be performed by optimizing the point spread function (PSF). The process of phase compensation based on the PSF optimization estimation phase error includes the following steps:

[0102] S701. First, due to the correlation of phase errors between different targets, a point target is manually specified as a reference in the imaging scene to ensure that both the reference target and the imaging target are observed by the radar. The round-trip propagation error between the two targets is expressed as:

[0103]

[0104] ΔR=|(ΔR1|-|ΔR2|)

[0105] The ideal antenna position for a single virtual antenna is (x′ i ,z i ′), and the actual antenna position includes the motion error Δz in the z-axis i , so the actual position is (x′ i ,z i ′+Δz i ), where ΔR1 and ΔR2 represent the round-trip propagation errors of target (x1, z1) and target (x2, z2), respectively, and ΔR represents the difference between ΔR1 and ΔR2.

[0106] Obviously, the smaller ΔR is, the closer the phase errors of the two targets are. According to the distribution of ΔR, a point target is set, and its estimated phase error can be very close to the phase error of the imaging target;

[0107] S702, then, the optimized PSF is obtained by fitting the phase history of the point target with the corresponding ideal quadratic phase history. Through this process, the phase error of the point target can be measured by quantifying the difference between the actual phase history of the point target and the ideal phase history. Using the principle of frequency modulated continuous wave (FMCW), the two targets in the range domain are separated by performing FFT along each reflected pulse. Since the phase history of the ideal trajectory point target follows a quadratic curve, the ideal phase change of the reference point target can be estimated by optimizing the following function:

[0108]

[0109] in, represents the estimated ideal phase change of the received signal of the reference point target at the i-th position, φ i is the corresponding disturbance phase change extracted by FFT, N is the number of pulses received, α, β and γ are the coefficients of the target curve, and the function finds the optimal α, β and γ values ​​that minimize the difference between the actual phase change of the reference object and the ideal phase change.

[0110] Once the estimated ideal phase history is obtained, the phase error caused by hand-held motion can be compensated as:

[0111]

[0112] in represents the motion error to be compensated, s i Represents motion errors that are corrupted by the received signal.

[0113] At this point, the phase in the signal has been corrected, which can better reflect the actual reflection characteristics of the target and improve the imaging quality.

[0114] S8, at this time, the signal after PSF optimization is obtained, that is, the motion error after phase error compensation The phase compensation of the radar signal uses the fast frequency domain near-field imaging method (RMA) to reconstruct the SAR image. The process of generating the corrected SAR image specifically includes the following steps:

[0115] S801, distance spectrum calculation: Perform one-dimensional fast Fourier transform on each pulse echo to obtain the distance spectrum, analyze the reflection characteristics of the target at different distances, and improve the accuracy of target detection and positioning:

[0116]

[0117] Where t represents time, x represents the antenna position, k r represents the distance frequency, S(k r ,x) represents the distance spectrum, s(t,x) represents the time domain signal, represents one-dimensional fast Fourier transform;

[0118] S802, azimuth spectrum calculation: Perform one-dimensional fast Fourier transform on each distance unit to obtain the azimuth spectrum, thereby improving the spatial resolution of the imaging system:

[0119]

[0120] Where S(k r ,kx ) is the azimuth spectrum, k x is the azimuth frequency;

[0121] S803, distance migration compensation: perform distance migration compensation (RMC) in the frequency domain to correct the distance offset caused by the change of the target position:

[0122]

[0123] Where S′(k r ,k x ) is the compensated frequency domain signal;

[0124] S804, azimuth inverse Fourier transform: Perform one-dimensional inverse fast Fourier transform on the compensated frequency domain signal to obtain the compensated distance spectrum, maintaining the integrity and accuracy of the signal:

[0125]

[0126] Where S′(k r ,x) represents the distance spectrum after compensation;

[0127] S805, distance inverse Fourier transform: perform one-dimensional inverse fast Fourier transform on the compensated distance spectrum to obtain the imaging result:

[0128]

[0129] Where s′(t,x) represents the compensated time domain signal.

[0130] S9. After obtaining the high-precision target human body signal, perform signal processing calculation to obtain the phase change of the human body signal. The specific steps are as follows:

[0131] S901. Data preprocessing: For the read raw data, some data preprocessing operations need to be performed, including data format conversion, correction, and data reorganization, so as to make the data clearer and more accurate for subsequent analysis.

[0132] S902, distance dimension FFT: Perform FFT on a single beat frequency signal to obtain the spectrum of the beat frequency signal. The calculated spectrum peak corresponds to the distance of the measured target, which is called distance dimension FFT. The distance dimension FFT transform improves the distance measurement accuracy and helps to suppress noise and interference.

[0133] The specific formula for the relationship between the maximum detection distance of FMCW radar and the sampling rate of ADC is:

[0134]

[0135] where d maxIndicates the maximum detection distance of the radar, F s represents the sampling rate, c represents the speed of light, and S represents the slope of the linear frequency modulation pulse in the frequency domain diagram.

[0136] The spectrum peak, i.e., the distance corresponding to the detected target, can be obtained from the specific formula of the relationship between the maximum detection distance of the FMCW radar and the sampling rate of the ADC.

[0137] S903, extract the phase of the original signal: the data processed by steps S901 and S902 is first filtered through static clutter to remove some DC components, eliminate reflections from static objects that do not change with time, and amplify human body reflections. The change in phase corresponds to the composite displacement caused by chest expansion and contraction due to breathing and body vibrations due to heartbeat. It then looks at the phase of the RF wave related to the distance traveled, and the specific formula is:

[0138]

[0139] Where Φ(t) represents the phase of the signal, λ represents the wavelength, d(t) represents the travel distance, and t represents the time variable.

[0140] S904, phase unwrapping: Phase unwrapping is also called phase unwinding. Since the phase value is between [-π,π], phase unwrapping is required to obtain the actual displacement curve. Therefore, whenever the phase difference between consecutive values ​​is greater than or less than ±π, the phase unwrapping is obtained by subtracting 2π from the phase, such as Figure 2 As shown, by phase unwrapping, the 2π modular operation effect can be eliminated from the original phase signal to obtain a continuous phase curve.

[0141] S905, phase difference: by continuously subtracting the phase of the current sampling point from the previous sampling point, and performing phase difference operation on the expanded phase by subtracting continuous phase values, it is beneficial to enhance the heartbeat signal and eliminate the phase drift existing in the hardware receiver, and suppress the respiratory signal and its harmonics.

[0142] S10. At this time, the phase difference signal after the above processing can be separated, but the influence of environmental noise on the signal needs to be removed. Therefore, the phase change obtained in step S9 is subjected to sliding average filtering and denoising, and then the breathing and heartbeat signals are separated by an IIR bandpass filter. Finally, the heart rate and breathing rate are calculated by the frequency domain FFT algorithm. The specific steps include:

[0143] S1001, sliding average filtering: before separating the phase difference signal obtained in step S9, a sliding average filtering algorithm is used to remove the influence of impulse noise caused by the test environment on the signal;

[0144] S1002, bandpass filter output respiratory signal and FFT spectrum analysis: Use IIR bandpass filter to process the phase signal obtained above, perform fast Fourier transform on the respiratory signal separated by the bandpass filter to obtain the respiratory frequency, and then convert it into respiratory rate. The specific formula is:

[0145] RR=f RR ×60(times / min)

[0146] where f RR represents respiratory frequency, RR represents respiratory rate;

[0147] S1003, band-pass filter output heartbeat signal and FFT spectrum analysis: Perform fast Fourier transform on the heartbeat signal separated by the band-pass filter to obtain the heartbeat frequency, which is then converted into the heartbeat rate. The specific formula is:

[0148] HR = f HR ×60(times / min)

[0149] where f HR represents respiratory rate and HR represents respiratory rate.

[0150] like Figure 3 As shown, Figure 3 (a) is the respiration time domain waveform after processing, and (b) is the heartbeat time domain waveform after processing. Both are within the range of normal human respiration and heartbeat, indicating that the processing method is effective.

[0151] S11. At this point, the heartbeat and breathing signals of the target person have been obtained. Next, it is necessary to extract emotion-related features from the signals and then analyze the emotions. Therefore, feature extraction is performed on the heartbeat and breathing signals obtained from step S10. These features can be divided into time domain analysis, frequency domain analysis, time-frequency analysis, Poincaré diagram, sample entropy and detrended fluctuation analysis, thereby ignoring information irrelevant to the target, reducing the amount of calculation, and improving the generalization ability of the model.

[0152] S12: Currently, only relevant features related to emotions can be obtained, and the corresponding emotions of the target person cannot be obtained. Therefore, in order to obtain the emotions of the target person, this embodiment uses the SVM model to perform emotion recognition.

[0153] In this embodiment, the SVM model uses relevant data in the MAHNOB-HCI-Tagging Database public database, and the process of training the model includes the following steps:

[0154] S1201, extracting training data: using relevant data in the MAHNOB-HCI-Tagging Database public database to calculate the corresponding features in the heartbeat and breathing signals, and extracting the corresponding valence arousal index and emotion classification;

[0155] S1202, dividing the training set and the test set: 80% of the data is then divided for training, and 20% of the data is reserved for testing the training results;

[0156] S1203, defining an SVM model: using the training data divided in step S1202 to train an SVM model on valence and arousal, the model performs standardization on feature data during training, specifies a kernel function as a radial basis function, and automatically selects a scale parameter of the kernel function;

[0157] S1204, training SVM model: based on the Error-Correcting Output Codes (ECOC) method, training a multi-class classification model, using the previously defined SVM template as the basic classifier, inputting the features and labels of the training set for model training;

[0158] S1205, observe the training effect: use the test data divided by step S1202 to test the SVM model trained by step S1203, and calculate the prediction accuracy of valence and arousal index.

[0159] The classification results of the SVM model in this embodiment are as follows Figure 4 As shown, the classification accuracy reaches 74.165%.

[0160] S13, inputting the relevant features calculated in step S11 into the trained SVM model to obtain the current emotion of the target person and complete emotion recognition. The specific steps include:

[0161] S1301 , inputting the relevant features calculated in step S11 into the trained SVM model to obtain the corresponding valence and arousal index.

[0162] S1302: Using the SVM model, convert the valence and arousal index obtained in step S1301 into emotions to obtain the current emotion of the target person.

[0163] S14, displaying the emotion identified in step S13. In this embodiment, the current emotion and the corresponding emotion suggestion are displayed in emoticons, such as: if the current emotion is anger, the user is advised to take a deep breath and look around to calm down.

[0164] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0165] Example 2

[0166] The present embodiment provides an emotion recognition robot based on millimeter wave radar, including a signal transmitting and receiving module, a point cloud conversion and static reflection elimination module, a dynamic point cloud grouping module, a centroid estimation and merging module, a target tracking module, a human body reflection signal sending and receiving module, a phase error estimation and compensation module, a SAR image reconstruction module, a small vibration displacement change acquisition module, a heartbeat and breathing signal separation module, a feature extraction module, a model training module, an emotion recognition module, a motor module, an LCD display module, an ESP8266 communication module, a single-chip microcomputer module, a robot shell module and a computer program signal that can be run on a processor. Among them, the transmitting and receiving module includes a millimeter-wave radar, which is used to use the millimeter-wave radar to send the modulated millimeter-wave signal as a transmitting signal to the surrounding environment, and receive the reflected echo signal, and obtain the intermediate frequency signal after mixing as the received data; the point cloud conversion and static reflection elimination module is used to convert the received data into a point cloud format, remove static reflections, and obtain dynamic point cloud data and timestamps of the surrounding environment; the dynamic point cloud grouping module is used to group the dynamic point cloud data into clusters; the centroid estimation and merging module obtains the centroid estimation value of the target based on the dynamic point cloud data of each cluster, and converts the point cloud from the same object into a single point observation with associated uncertainty; the target tracking module is used to track the centroid estimation value and determine the position of the target person at each moment; the human body reflection signal sending and receiving module is used to send the millimeter wave signal to the target human body through the robot's uniform horizontal movement, and receive the evenly spaced echo signals reflected by the human body. ; The phase error estimation and compensation module is used to optimize the point spread function of the uniformly spaced echo signal to achieve phase compensation; the SAR image reconstruction module is used to use the fast frequency domain near-field imaging method to perform SAR image reconstruction correction to generate a corrected SAR image; the small vibration displacement change acquisition module is used to extract the phase change of the target distance human body signal from the human body reflection signal obtained by the SAR image reconstruction module, obtain the small vibration displacement change, and then obtain the phase difference signal; the heartbeat and breathing signal separation module is used to process the phase change obtained in the small vibration displacement change acquisition module through an IIR bandpass filter to separate the breathing and heartbeat signals of the target person; the feature extraction module is used to extract emotion-related features from the breathing and heartbeat signals; the model training module is used to train the emotion recognition model. In this embodiment, the SVM model is used, and the data in the MAHNOB-HCI-Tagging Database public database is used to train the model; the emotion recognition module is used to input the emotion-related features obtained in the feature extraction module into the trained emotion recognition model to obtain the current emotion of the target person; the motor module is used to control the movement of the robot, which is convenient for locating human targets and improving accuracy by using the RMA algorithm through PSF optimization;The LCD display module is used to display short-cycle animations corresponding to emotions, and when the emotions change, after the current animation cycle is completed, the animation transition is performed, switching to another emoticon display, and more active human-computer interaction functions are added through the TFT touch screen; the communication module is used to receive information from the cloud server and change the emotional state based on the received information. In this embodiment, the ESP8266 communication module is used to receive information from conventional Wifi and hotspot devices; the single-chip computer module is connected to the communication module for LCD screen control, motor control, user perception and communication task control. The user perception and motor control clocks maintain a lower priority and always run to adjust the posture. When the emotional state is received, the LCD screen emoticon animation is controlled to change; the robot shell module is used to install the millimeter wave radar board, motor module, single-chip computer module, LCD display module and communication module to achieve compact optimization of the robot's internal structure and exquisite appearance. ;

[0167] It should be noted that each module in the above system corresponds to the specific steps of the method provided in Example 1 of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in Example 1 of the present invention.

[0168] In this embodiment, the test scenario is that the user is far away from the emotion recognition robot indoors and is not facing the robot, and the chest and the emotion recognition robot are on the same horizontal plane. The millimeter wave signal is emitted from the TI IWR1443 millimeter wave radar, which is a 76-81GHz FMCW millimeter wave signal. The target person watches a video clip that triggers a specific emotional state.

[0169] Specifically, in this embodiment, a total of 3 volunteers participated in the test as target persons. The chest of the volunteers was located at the same level as the emotion recognition robot, 5 meters away and not facing the emotion recognition robot. During the test, they could be standing, sitting or walking to measure different emotions multiple times. The final detection result had an accuracy rate of about 72%.

[0170] In another embodiment, the emotion recognition robot can be connected to a cloud server, and the radar emission signal and echo signal collected by the emotion recognition robot are uploaded to the cloud server, and the cloud server processes the recognized emotions according to the method provided in Example 1 of the present invention, and then feeds back the recognized emotions to the display screen of the robot. In this embodiment, the cloud server is configured as an Alibaba Cloud server with ECS-ARM architecture, Yitian c8y, 2 cores and 4GB.

[0171] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for emotion recognition based on millimeter wave radar, characterized in that: The following steps are involved: Receive the transmission signal and the corresponding echo signal of the millimeter-wave radar, mix them to obtain the intermediate frequency signal as the received data, convert the received data into point cloud format, remove the static reflection, and obtain the dynamic point cloud data and timestamp of the surrounding environment; Grouping the dynamic point cloud data into clusters; Obtaining a centroid estimate of the target based on each cluster of dynamic point cloud data, tracking the centroid estimate, and determining the position of the target person at each moment; Send a millimeter wave signal moving horizontally at a uniform speed to a target person, and receive evenly spaced echo signals reflected by the target person; Performing point spread function optimization on the evenly spaced echo signals, and performing SAR image reconstruction correction using a fast frequency domain near-field imaging method; The phase change of the human body signal is calculated based on the reconstructed and corrected SAR image to obtain the phase difference signal, and the phase difference signal is separated to obtain the breathing and heartbeat signals of the target person; Emotion-related features are extracted from the breathing and heartbeat signals, and the current emotion of the target person is obtained based on the emotion-related features.

2. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: Grouping the dynamic point cloud data into clusters specifically includes: Using Mahalanobis distance as the metric, DBSCAN algorithm is applied to group the dynamic point cloud data into clusters.

3. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The process of obtaining the estimated value of the centroid includes: Calculate the measurement noise covariance matrix corresponding to each cluster of point cloud data; The centroid position is calculated based on the sum of the products of the inverse of the measurement noise covariance matrix and the measurement value of each point in each cluster point cloud data, and the noise covariance matrix of the centroid is obtained according to the inverse of the sum of the inverses of all measurement noise covariance matrices.

4. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: Tracking the centroid estimate using a joint integrated probabilistic data association algorithm includes the following steps: Initialize the target state using the extended Kalman filter; Use the joint integrated probabilistic data association algorithm to calculate the association probability and associate the new observation data with the existing trajectories; Update the target's state estimate using an extended Kalman filter; Determine whether to confirm or delete the track based on the probability of the target's existence.

5. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The point spread function optimization specifically includes: A point target is designated as a reference target in the imaging scene, and the round-trip propagation error between the reference target and the imaging target is calculated; Based on the difference distribution of the round-trip propagation error between different point targets, a final point target is set, and phase error compensation is obtained based on the final point target phase history and the corresponding ideal quadratic phase history to obtain a signal after point spread function optimization.

6. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The SAR image reconstruction correction specifically includes: Performing one-dimensional fast Fourier transform on each pulse echo in the signal after point spread function optimization to obtain a range spectrum, and performing one-dimensional fast Fourier transform on each range spectrum to obtain an azimuth spectrum; Performing distance migration compensation based on the azimuth spectrum in the frequency domain to obtain a compensated frequency domain signal; The compensated frequency domain signal is subjected to one-dimensional inverse fast Fourier transform to obtain a compensated range spectrum, and the compensated range spectrum is subjected to one-dimensional inverse fast Fourier transform to obtain an imaging result.

7. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The process of obtaining the phase change of the human body signal includes: Read the raw data from the reconstructed and corrected SAR image and perform data preprocessing; Extract the original phase signal, and perform phase unwrapping whenever the phase difference between consecutive values ​​is greater than or less than ±π to obtain a continuous phase curve; A phase difference operation is performed on the unwrapped phase to obtain a phase difference signal.

8. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The separation of the phase difference signal specifically includes: The sliding average filtering algorithm is used to denoise the phase difference signal; The denoised signal is bandpass filtered to separate the breathing and heartbeat signals of the target person.

9. The emotion recognition method based on millimeter wave radar according to claim 1, characterized in that: The emotion-related features include time domain analysis, frequency domain analysis, time-frequency analysis, Poincaré diagram, sample entropy and detrended fluctuation analysis. The current emotion of the target person is obtained based on the emotion-related feature recognition through the SVM model.

10. An emotion recognition robot based on millimeter wave radar, characterized in that: include: The signal transmitting and receiving module includes a millimeter wave radar, which is used to use the millimeter wave radar to send the modulated millimeter wave signal as a transmitting signal to the surrounding environment, and receive the reflected echo signal, and obtain an intermediate frequency signal after mixing as receiving data; Point cloud conversion and static reflection elimination module, used to convert the received data into point cloud format, remove static reflections, and obtain dynamic point cloud data and timestamp of the surrounding environment; A dynamic point cloud grouping module, used for grouping the dynamic point cloud data into clusters; The centroid estimation and merging module obtains the centroid estimation value of the target based on the dynamic point cloud data of each cluster; A target tracking module, used to track the center of mass estimation value to determine the position of the target person at each moment; The human body reflection signal sending and receiving module is used to send millimeter wave signals to the target human body through the robot's uniform horizontal movement, and receive the evenly spaced echo signals reflected by the human body; A phase error estimation and compensation module, used for optimizing the point spread function of the evenly spaced echo signals to achieve phase compensation; A SAR image reconstruction module is used to perform SAR image reconstruction correction using a fast frequency domain near-field imaging method to generate a corrected SAR image; A small vibration displacement change acquisition module is used to extract the phase change of the target distance human body signal from the human body reflection signal obtained by the SAR image reconstruction module to obtain a phase difference signal; The heartbeat and breathing signal separation module is used to separate the phase difference signal to obtain the breathing and heartbeat signals of the target person; A feature extraction module, used to extract emotion-related features from the breathing and heartbeat signals; Model training module, used to train emotion recognition model; The emotion recognition module is used to input the emotion-related features obtained in the feature extraction module into the trained emotion recognition model to obtain the current emotion of the target person; Motor module, used to control the movement of the robot; LCD display module: used to display short-cycle animations corresponding to emotions. When the emotion changes, the animation transition is performed after the current animation cycle is completed, switching to another display mode, and providing human-computer interaction functions; Communication module: used to receive information from the cloud server and change the emotional state based on the received information; The single chip microcomputer module is connected with the communication module and is used to realize the control of the motor module and the LCD display module; The robot shell module is used to install the millimeter wave radar board, motor module, single-chip computer module, LCD display module and communication module.

Images