Face key point detection system and method based on millimeter wave radar

Abstract

The application discloses a kind of face key point detection system and method based on millimeter wave radar, system includes millimeter wave data acquisition module, user head detection module, face feature extraction module and face key point detection module;Through millimeter wave data acquisition module, millimeter wave data is collected, and horizontal movement and vertical movement of user face muscle are simultaneously captured;The position of user face on distance-angle chart is determined by user head detection module;Then, through face feature extraction module, the interference generated by natural body swing is eliminated, and the phase information is compensated, to extract effective face movement features;Finally, face key point is detected by face key point detection module.The application realizes high-precision face key point detection based on millimeter wave radar, and provides basic technical support for subsequent expression recognition, fatigue monitoring, augmented reality and various human-computer interaction applications.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of millimeter wave wireless sensing and human-computer interaction, and particularly relates to a face key point detection system and method based on a millimeter wave radar. BACKGROUND

[0002] As one of the most common and intuitive facial feature representations, face key points have been widely used in various human-computer interaction applications such as expression recognition, fatigue monitoring, augmented reality, and virtual reality. These applications rely on high-precision face key point detection results, otherwise their performance will be significantly degraded. For example, if a face key point detection system cannot accurately identify the key points of the mouth, an expression recognition system will have difficulty in distinguishing the opening and closing of the user's mouth or the movement of the corners of the mouth, which may lead to incorrect expression recognition results. Therefore, high-precision face key point systems and methods are crucial to the performance of these applications.

[0003] Current main face key point detection solutions include the following:

[0004] (1) Computer vision-based detection methods: This type of method usually uses deep neural networks to identify face key points in images, but due to the limitations of the camera, this type of method has the risk of leaking user privacy, is very sensitive to lighting conditions, and is easily affected by face accessories such as glasses and masks.

[0005] (2) Wearable device-based detection methods: This type of method usually relies on special devices to read the user's facial electromyographic signals to infer the positions of face key points. However, due to the limitations of sensing accuracy, these methods are difficult to accurately detect face key points. In addition, the user's head orientation cannot be sensed, further limiting its scope of application in practical applications.

[0006] (3) Millimeter wave radar-based detection methods: Traditional millimeter wave radar-based detection methods have relatively low spatial resolution, while methods based on synthetic aperture radar algorithms can achieve high spatial resolution, but have great challenges in terms of time efficiency.

[0007] (4) Detection methods based on other wireless devices: Using RFID (Radio Frequency Identification), Wi-Fi devices, or ultrasonic devices for face state sensing, these methods have limited spatial resolution and cannot achieve accurate detection of face key points.

[0008] Therefore, based on the above considerations, it is necessary to propose a face key point detection system and method based on a millimeter wave radar, which can balance spatial resolution and time efficiency, thereby achieving efficient and accurate face key point detection. SUMMARY

[0009] In view of the above problems of the prior art, the present application aims to provide a millimeter wave radar-based face key point detection system and method to solve the problems of the prior art, such as the privacy leakage of cameras, the sensitivity to light conditions, the inability to penetrate the occlusion, and the inability of most wireless device-based detection methods to achieve sufficient resolution or to balance spatial resolution and time efficiency.

[0010] To achieve the above-mentioned purposes, the technical solutions adopted by the present application are as follows:

[0011] The millimeter wave radar-based face key point detection system of the present application comprises a millimeter wave data acquisition module, a user head detection module, a face feature extraction module, and a face key point detection module.

[0012] The millimeter wave data acquisition module comprises a millimeter wave radar and a data transmission unit.

[0013] The millimeter wave radar is used to transmit radio frequency signals and receive echo signals reflected by the human body, and to generate millimeter wave intermediate frequency signal data through mixing processing.

[0014] The data transmission unit is used to send the millimeter wave intermediate frequency signal data to the user head detection module.

[0015] The user head detection module is used to convert the millimeter wave intermediate frequency signal data into a range-angle graph through 3D Fourier transform, to determine the distance range where the user's face is located, to dynamically adjust the speed resolution of the millimeter wave radar to generate a super-resolution speed-angle graph, to determine the angle range where the user's face is located in the super-resolution speed-angle graph by using the speed difference between the user's head and body, and to determine the position of the face on the range-angle graph in combination with the distance range and the angle range where the user's face is located, and to perform real-time tracking.

[0016] The face feature extraction module is used to extract phase information from the face region, to eliminate the interference caused by natural body swaying through the face-body reflection principle, and to compensate the phase information through the face symmetry principle to extract effective face motion features.

[0017] The face key point detection module is used to input the effective face motion features as a neural network, and to detect the face key point position by using a spatial-temporal fusion network (STFN) based on a spatial-temporal attention mechanism.

[0018] Further, the millimeter wave radar is arranged in a tilt manner, so that the millimeter wave radar has a higher spatial resolution.

[0019] Further, the tilt arrangement of the millimeter wave radar is specifically as follows:

[0020] (11) The millimeter wave radar is rotated by 90° around the Z axis of the millimeter wave coordinate system, so that the direction of the millimeter wave antenna array is parallel to the vertical direction to capture the vertical movement of the facial muscles of the user;

[0021] (12) The millimeter wave radar is rotated by 20° to 40° around the Y axis of the millimeter wave coordinate system, so that it is located in the oblique front of the user, and the millimeter wave propagation direction is oblique to the face of the user, to capture the horizontal movement of the facial muscles of the user.

[0022] Further, the user head detection module determines the distance range where the face of the user is located on the range-angle graph through a constant false alarm rate (CFAR) detection algorithm, and the specific steps are as follows:

[0023] (21) The false alarm rate of the constant false alarm rate detection algorithm is set to p;

[0024] (22) The energy threshold of each point on the range-angle graph is calculated based on the false alarm rate p of the constant false alarm rate detection algorithm, and the point with an actual energy greater than the energy threshold is taken as a detection target;

[0025] (23) The distance range [R min ,R max ] of the detection target is counted; and R min , is taken as the distance range where the face is located.

[0026] Further, the user head detection module dynamically adjusts the speed resolution of the millimeter wave radar through a Doppler super-resolution algorithm, and the Doppler super-resolution algorithm is specifically as follows:

[0027] (31) The type of the mixed millimeter wave data is set to [N f ,N c ,NT X ,N RX ,N s ], where N f is the total number of millimeter wave data frames, N c is the number of chirps contained in each millimeter wave data frame, N TX and N RX are the number of transmitting antennas and the number of receiving antennas of the millimeter wave radar respectively, and N s is the number of samples on each chirp;

[0028] (32) Transform the original millimeter wave data into reorganized millimeter wave data by the method of sliding window, wherein the length of the sliding window is equal to the Doppler factor N DF , and the step is 1; the Doppler factor N DF is a parameter set by the head positioning algorithm, and the specific operation of reorganizing the millimeter wave data is to reorganize [N f , N c , N TX , N RX , N s ] of the original millimeter wave data into [N f ×N c , N TX ×N RX , N s ];

[0029] (33) Discrete Fourier transform the reorganized millimeter wave data along the fast time dimension (i.e. the last dimension), and multiply the distance resolution d res to obtain the distance information of the objects in the scene; use a peak detection algorithm to detect the peak region in the distance dimension, and the peak region corresponds to the distance range of the user's head from the millimeter wave radar;

[0030] (34) Take the reorganized millimeter wave data corresponding to the median of the distance range of the user's head, and perform discrete Fourier transform along the slow time dimension (i.e. the first dimension) and the antenna dimension (i.e. the second dimension); multiply the slow time dimension by the velocity resolution v res to obtain the velocity information; multiply the antenna dimension by the angle resolution θ res to obtain the angle information; and the final processing result is a super-resolution velocity-angle map.

[0031] Further, the specific steps of dynamically adjusting the velocity resolution of the millimeter wave radar by the Doppler super-resolution algorithm and determining the angle range in which the user's face is located in the super-resolution velocity-angle map include:

[0032] (41) Set the initial value of the Doppler factor N DF , the velocity significance threshold ∈, and the maximum value of the Doppler factor N

[0033] (42) Convert the millimeter wave data into a super-resolution velocity-angle map by the Doppler super-resolution algorithm;

[0034] (43) Detect the target in the super-resolution velocity-angle map by a constant false alarm rate detection algorithm; and cluster the detected targets into clusters by a density-based spatial clustering of applications with noise (DBSCAN) algorithm; and record the angle range in which the clusters are located.

[0035] (44) According to Calculate the velocity significance VSI, where N clu represents the number of clusters obtained by density-based spatial clustering of applications with noise, and respectively represent the maximum speed and the minimum speed (which can be negative) of the i-th cluster, and respectively represent the maximum speed and the minimum speed (which can be negative) of all clusters;

[0036] (45) Determine whether the velocity significance VSI is less than the velocity significance threshold ∈, if VSI<∈, then let the Doppler factor N DF =N DF ×2, repeat steps (42)-(44) until VSI≥∈ or

[0037] (46) The angle range of the cluster obtained by the density-based spatial clustering of applications with noise recorded is taken as the angle range of the user's face.

[0038] Further, the face-body reflection principle is specifically:

[0039] (51) Set the correlation coefficient k of the phase influence of the natural body swing on the face area reflection signal and the phase influence of the body area reflection signal;

[0040] (52) Extract the phase information of the face area reflection signal and the phase information of the reflection signal of the body area b (t) = τ b +kΔτ bsway (t), where, is the initial phase of the face area reflection signal of the i-th distance-angle bin, τ b is the initial phase of the body area reflection signal, is the phase influence of the face movement on the face area reflection signal of the i-th distance-angle bin, Δτ bsway (t) is the phase influence of the natural body swing on the face area reflection signal;

[0041] (53) Calculate the angle corresponding to the i-th distance-angle bin and the angle of the body area corresponding to the bin, then the correlation coefficient

[0042] (54) Combine steps (52) and (53) to obtain the phase information after interference elimination

[0043] Further, the face symmetry principle is specifically:

[0044] (61) First-order difference of phase information of the face region is calculated, and the displacement of each face bin is calculated according to the difference result

[0045] (62) The angle between the face motion and the face orientation is calculated by the least square method As follows:

[0046]

[0047] Wherein, and are the displacements of the i-th symmetric bin of the left face and the right face respectively, N sym is the total number of symmetric bins;

[0048] (63) The outliers in the calculation process of step (61) are judged by the RANSAC algorithm, and are marked as asymmetric regions, and the remaining face bins are marked as symmetric regions;

[0049] (64) If the millimeter wave radar is deployed on the left side of the face, the phase compensation required for each bin on the right side of the face is calculated; if the bin on the right side of the face belongs to the symmetric region, the first-order difference of the compensated phase is Wherein, ΔR l is the displacement measured at the bin of the left face in the symmetric position of the bin on the right side of the face; if the bin on the right side of the face belongs to the asymmetric region, the first-order difference of the compensated phase is Wherein, ΔR r is the displacement measured at the bin.

[0050] Further, the network structure of the space-time fusion network based on the space-time attention mechanism is sequentially from the input end to the output end: layer normalization layer, space-time attention layer, cascade layer, layer normalization layer and linear layer; wherein, the space-time attention layer contains two branches, each branch is 1 68 attention layer connected with 1 linear layer; the space-time fusion network based on the space-time attention mechanism takes the face motion feature of type [N f ×N c ,N bin ] as input, N f is the total number of millimeter wave data frames, N c is the number of chirps contained in each millimeter wave data, N binThe number of face bins; after the face motion feature is token embedded and position coded, it is input into the network, and the attention weight is calculated along two dimensions of the face motion feature.

[0051] The face key point detection method based on the millimeter wave radar, based on the above system, includes a calibration stage and a detection stage.

[0052] Calibration stage:

[0053] 11) The millimeter wave radar is deployed in the scene in a tilted deployment manner, and a user who keeps natural head shaking in the scene is detected;

[0054] 12) The millimeter wave radar receives the echo signal reflected by the human body, and generates millimeter wave intermediate frequency signal data through mixing processing;

[0055] 13) The millimeter wave intermediate frequency signal data is converted into a distance-angle graph through 3D Fourier transform; the distance range where the face is located is determined on the distance-angle graph;

[0056] 14) The speed resolution of the millimeter wave radar is dynamically adjusted, and the angle range where the face is located is determined in the super-resolution speed-angle graph;

[0057] 15) The initial position of the face on the distance-angle graph is determined in combination with the distance range and the angle range where the face is located, and the historical trajectory information of the face position is recorded;

[0058] Detection stage:

[0059] 21) The millimeter wave radar is deployed in the scene in a tilted deployment manner, and a user who keeps natural head shaking in the scene is detected;

[0060] 22) The millimeter wave radar receives the echo signal reflected by the human body, and generates millimeter wave intermediate frequency signal data through mixing processing;

[0061] 23) The millimeter wave intermediate frequency signal data is converted into a distance-angle graph through 3D Fourier transform; the distance range where the face is located is determined on the distance-angle graph;

[0062] 24) The speed resolution of the millimeter wave radar is dynamically adjusted, and the angle range where the face is located is determined in the super-resolution speed-angle graph;

[0063] 25) The position of the face on the distance-angle graph is determined in combination with the distance range and the angle range where the face is located, as the measured position of the face at the current time;

[0064] 26) the measured position of the face at the current time and the historical trajectory information are taken as inputs of the Kalman filter to obtain the final position estimation of the face; and the measured position of the face at the current time is added to the historical trajectory information of the face position;

[0065] 27) phase information of the region corresponding to the face position is extracted; interference caused by natural body sway is eliminated, and the phase information is compensated to extract effective face movement features;

[0066] 28) the face movement features are taken as inputs of a neural network, and a space-time fusion network based on a space-time attention mechanism is used to detect the face key point positions.

[0067] The beneficial effects of the present application are:

[0068] 1. The present application can capture the horizontal movement and vertical movement of the user's facial muscles respectively, improve the integrity of the face movement feature extraction, and obtain a high spatial resolution without using a synthetic aperture radar algorithm.

[0069] 2. The present application can calculate the position of the face on the range-angle graph, and then distinguish the face reflection signal and the body reflection signal in space, avoiding the interference caused by the mixing of the face reflection signal and the body reflection signal.

[0070] 3. The present application eliminates the interference caused by natural body sway, and compensates the phase of the reflection signal with weak energy in the detection process, thereby enhancing the signal-to-noise ratio of the face movement features and improving the accuracy of the face key point detection results.

[0071] 4. The present application realizes an end-to-end face key point detection neural network based on millimeter wave data, and the average error of the face key point is only 2.81 mm.

[0072] 5. The present application realizes high-precision face key point detection based on millimeter wave radar, provides a basic technical support for subsequent expression recognition, fatigue monitoring, augmented reality and other various human-computer interaction applications, and has low environmental requirements, is not easy to be disturbed, has high robustness, and can work normally in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 It is the architecture diagram of the face key point detection system of the present application;

[0074] Figure 2 It is a schematic diagram of the tilt deployment of the millimeter wave radar of the present application;

[0075] Figure 3a It is a schematic diagram of the range-angle graph after 3D Fourier transform of the present application;

[0076] Figure 3b A schematic diagram of a super-resolution velocity-angle map generated by the Doppler super-resolution algorithm of the present application;

[0077] Figure 4 A schematic diagram of the face-body reflection principle of the present application;

[0078] Figure 5 A schematic diagram of the face symmetry principle of the present application;

[0079] Figure 6 A schematic diagram of the network structure of the space-time fusion network of the present application;

[0080] Figure 7 A flowchart of the method of the present application. DETAILED DESCRIPTION

[0081] For the convenience of those skilled in the art, the present application will be further described below in conjunction with the embodiments and the accompanying drawings, and the content mentioned in the embodiments is not a limitation of the present application.

[0082] Referring to Figures 1 to 6 The face key point detection system based on a millimeter wave radar of the present application comprises a millimeter wave data acquisition module, a user head detection module, a face feature extraction module and a face key point detection module.

[0083] The millimeter wave data acquisition module comprises a millimeter wave radar and a data transmission unit.

[0084] The millimeter wave radar is used for transmitting wireless radio frequency signals and receiving echo signals reflected by the human body, and generates millimeter wave intermediate frequency signal data through mixing processing.

[0085] The data transmission unit is used for sending the millimeter wave intermediate frequency signal data to the user head detection module.

[0086] Specifically, the millimeter wave radar is arranged in an inclined manner, so that the millimeter wave radar has higher spatial resolution.

[0087] The inclined arrangement of the millimeter wave radar is specifically as follows:

[0088] (11) The millimeter wave radar is rotated by 90° around the Z axis of the millimeter wave coordinate system, so that the direction of the millimeter wave antenna array is parallel to the vertical direction to capture the vertical movement of the user's facial muscles.

[0089] (12) The millimeter wave radar is rotated by 20° to 40° around the Y axis of the millimeter wave coordinate system, so that it is located in the oblique front of the user, and the millimeter wave propagation direction is oblique to the user's face, to capture the horizontal movement of the user's facial muscles.

[0090] The user head detection module is configured to convert millimeter wave intermediate frequency signal data into a distance-angle graph through 3D Fourier transform; determine a distance range in which a user's face is located on the distance-angle graph through a constant false alarm rate (CFAR) detection algorithm; dynamically adjust a speed resolution of the millimeter wave radar through a Doppler super-resolution algorithm to generate a super-resolution speed-angle graph; determine an angle range in which the user's face is located in the super-resolution speed-angle graph by using a speed difference between the user's head and body; and determine a position of the face on the distance-angle graph in combination with the distance range and the angle range in which the user's face is located, and perform real-time tracking.

[0091] Specifically, the user head detection module determines a distance range in which a user's face is located on a distance-angle graph through a constant false alarm rate (CFAR) detection algorithm, and the specific steps are as follows:

[0092] (21) Set a false alarm rate of the constant false alarm rate detection algorithm as p;

[0093] (22) Calculate an energy threshold of each point on the distance-angle graph based on the false alarm rate p of the constant false alarm rate detection algorithm, and take a point with an actual energy greater than the energy threshold as a detection target;

[0094] (23) Statistically determine a distance range [R min ,R max ] of the detection target; and take R min , as the distance range in which the face is located.

[0095] The Doppler super-resolution algorithm is specifically as follows:

[0096] (31) Set a mixed millimeter wave data type as [N f ,N c ,N TX ,N RX ,N s ], where N f is a total number of millimeter wave data frames, N c is a number of chirps included in each millimeter wave data frame, N TX and N RX are respectively a number of transmitting antennas and a number of receiving antennas of the millimeter wave radar, and N s is a sampling number on each chirp;

[0097] (32) Transform original millimeter wave data into recombined millimeter wave data in a sliding window method, where a sliding window length is equal to a Doppler factor N DF , and a step length is 1; and the Doppler factor N DFParameters set for the head positioning algorithm, the specific operation of the millimeter wave data reorganization is to reorganize the original millimeter wave data [N f ,N c ,N TX ,N RX ,N s ] into [N f ×N c ,N TX ×N RX ,N s ];

[0098] (33) Discrete Fourier transform is performed on the reorganized millimeter wave data along the fast time dimension (i.e. the last dimension), and then multiplied by the distance resolution d res , to obtain the distance information of the objects in the scene; the peak value region in the distance dimension is detected by using a peak value detection algorithm, and the peak value region corresponds to the distance range of the user's head from the millimeter wave radar;

[0099] (34) The reorganized millimeter wave data corresponding to the middle value in the distance range of the user's head is taken, and discrete Fourier transform is performed on the slow time dimension (i.e. the first dimension) and the antenna dimension (i.e. the second dimension) thereof; the slow time dimension is multiplied by the velocity resolution v res , to correspond to the velocity information; the antenna dimension is multiplied by the angle resolution θ res , to correspond to the angle information; and the final processing result is a super-resolution velocity-angle map.

[0100] The specific steps of dynamically adjusting the velocity resolution of the millimeter wave radar by using the Doppler super-resolution algorithm and determining the angle range in which the user's face is located in the super-resolution velocity-angle map include:

[0101] (41) Setting the Doppler factor N DF , the velocity significance threshold ∈, and the maximum value of the Doppler factor

[0102] (42) Converting the millimeter wave data into a super-resolution velocity-angle map by using the Doppler super-resolution algorithm;

[0103] (43) Detecting the target in the super-resolution velocity-angle map by using a constant false alarm rate detection algorithm; and clustering the detected target into clusters by using a density-based spatial clustering of applications with noise (DBSCAN) algorithm; and recording the angle range in which the clusters are located;

[0104] (44) Calculating the velocity significance VSI according to , wherein N cludenotes the number of clusters obtained by density-based spatial clustering of applications with noise, and denote the maximum and minimum speed (may be negative) of the i-th cluster, respectively, and denote the maximum and minimum speed (may be negative) of all clusters, respectively;

[0105] (45) judge whether the speed significance VSI is less than the speed significance threshold ∈, if VSI < ∈, then let the Doppler factor N DF = N DF × 2, repeat steps (42)-(44) until VSI ≥ ∈ or

[0106] (46) take the angle range of the recorded cluster obtained by the density-based spatial clustering of applications with noise as the angle range of the user's face.

[0107] The face feature extraction module is configured to extract phase information from the face region, eliminate interference caused by natural body sway through the face-body reflection principle, and then compensate the phase information through the face symmetry principle to extract effective face motion features.

[0108] The face-body reflection principle comprises:

[0109] (51) set a correlation coefficient k of the phase influence of the natural body sway on the face region reflection signal and the phase influence of the body region reflection signal;

[0110] (52) extract the phase information of the face region reflection signal and the phase information of the body region reflection signal b (t) = τ b +kΔτ bsway (t), wherein, is the initial phase of the face region reflection signal of the i-th distance-angle bin, τ b is the initial phase of the body region reflection signal, is the phase influence of the face motion on the face region reflection signal of the i-th distance-angle bin, Δτ bsway (t) is the phase influence of the natural body sway on the face region reflection signal;

[0111] (53) calculate the angle corresponding to the i-th distance-angle bin and the angle corresponding to the bin of the body region,

[0112] (54) Simultaneously step (52) and step (53) to obtain the phase information after interference cancellation

[0113] Wherein, the face symmetry principle is specifically:

[0114] (61) First-order difference ΔΦ is calculated for the phase information of the face region, and the displacement of each face bin is calculated according to the difference result

[0115] (62) The angle between the face motion and the face orientation is calculated by the least square method As follows:

[0116]

[0117] Wherein, And The displacement of the i-th symmetric bin of the left and right faces respectively, N sym The total number of symmetric bins;

[0118] (63) The RANSAC algorithm is used to judge the outliers in the calculation process of step (61), and mark them as asymmetric regions, and the remaining face bins are marked as symmetric regions;

[0119] (64) If the millimeter wave radar is deployed on the left side of the face, then the phase compensation needed by each bin located on the right side of the face is calculated; if the bin located on the right side of the face belongs to the symmetric region, the first-order difference of the compensated phase is Wherein, ΔR l Is the displacement measured at the bin of the left face symmetric to the bin located on the right side of the face; if the bin located on the right side of the face belongs to the asymmetric region, the first-order difference of the compensated phase is Wherein, ΔR r Is the displacement measured at the bin.

[0120] The face key point detection module is used to take the effective face motion features as the neural network input, and uses the spatial-temporal fusion network (STFN) based on the spatial-temporal attention mechanism to detect the face key point position.

[0121] Specifically, the network structure of the spatio-temporal fusion network based on the spatio-temporal attention mechanism is sequentially from an input end to an output end: a layer normalization layer, a spatio-temporal attention layer, a cascade layer, a layer normalization layer and a linear layer; wherein the spatio-temporal attention layer contains two branches, and each branch is 1 68-head attention layer connected with 1 linear layer; the spatio-temporal fusion network based on the spatio-temporal attention mechanism takes a face motion feature with a form of [N f x N c , N bin ] as input, N f is a total number of millimeter wave data frames, N c is a number of chirps contained in each frame of millimeter wave data, N bin is a number of face bins; after the face motion feature is token embedded and position encoded, the face motion feature is input into the network, and attention weights are calculated along two dimensions of the face motion feature respectively.

[0122] Referring to Figure 7 , a face key point detection method based on a millimeter wave radar according to the present application is based on the above system and includes a calibration stage and a detection stage.

[0123] Calibration stage:

[0124] 11) The millimeter wave radar is deployed in the scene in a manner of inclined deployment, and a user who keeps natural head shaking in the scene is detected;

[0125] 12) The millimeter wave radar receives echo signals reflected by the human body, and generates millimeter wave intermediate frequency signal data through mixing processing;

[0126] 13) The millimeter wave intermediate frequency signal data is converted into a distance-angle graph through 3D Fourier transform; and a distance range where the face is located is determined on the distance-angle graph through a constant false alarm rate detection algorithm;

[0127] 14) The speed resolution of the millimeter wave radar is dynamically adjusted through a Doppler super-resolution algorithm, and an angle range where the face is located is determined in a super-resolution speed-angle graph;

[0128] 15) The initial position of the face on the distance-angle graph is determined in combination with the distance range and the angle range where the face is located, and the historical trajectory information of the face position is recorded.

[0129] Detection stage:

[0130] 21) The millimeter wave radar is deployed in the scene in a manner of inclined deployment, and a user who freely moves in the scene is detected;

[0131] 22) The millimeter wave radar receives echo signals reflected by the human body, and generates millimeter wave intermediate frequency signal data through mixing processing;

[0132] 23) Convert the millimeter wave intermediate frequency signal data into a distance-angle map by 3D Fourier transform; determine the distance range where the face is located on the distance-angle map by a constant false alarm rate detection algorithm;

[0133] 24) Dynamically adjust the speed resolution of the millimeter wave radar by a Doppler super-resolution algorithm, and determine the angle range where the face is located in the super-resolution speed-angle map;

[0134] 25) Determine the position of the face on the distance-angle map by combining the distance range and the angle range where the face is located, as the measured position of the face at the current time;

[0135] 26) Take the measured position of the face at the current time and the historical trajectory information as the input of the Kalman filter to obtain the final position estimation of the face; then add the measured position of the face at the current time to the historical trajectory information of the face position;

[0136] 27) Extract the phase information of the region corresponding to the face position; eliminate the interference caused by the natural shaking of the body by the face-body reflection principle, and then compensate the phase information by the face symmetry principle to extract the effective face motion feature;

[0137] 28) Take the face motion feature as the input of the neural network, and detect the face key point position by using a space-time fusion network based on a space-time attention mechanism.

[0138] The present application has many specific application ways, and the above description is only the preferred embodiment of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the principle of the present application, some improvements can be made, and these improvements should also be considered as the protection scope of the present application.

Claims

1. A millimeter wave radar based face keypoint detection system, characterized in that, The application relates to a millimeter wave data acquisition module, a user head detection module, a face feature extraction module and a face key point detection module. The millimeter wave data acquisition module comprises a millimeter wave radar and a data transmission unit. The millimeter wave radar is used for transmitting wireless radio frequency signals and receiving echo signals reflected by a human body, and millimeter wave intermediate frequency signal data are generated through frequency mixing processing. The data transmission unit is used for sending the millimeter wave intermediate frequency signal data to the user head detection module. The user head detection module is used for converting the millimeter wave intermediate frequency signal data into a distance-angle graph through 3D Fourier transform; determining a distance range where a user's face is located; dynamically adjusting the speed resolution of the millimeter wave radar to generate a super-resolution speed-angle graph; determining an angle range where the user's face is located in the super-resolution speed-angle graph by using the speed difference between the user's head and body; and determining the position of the face on the distance-angle graph in combination with the distance range and the angle range where the user's face is located, and performing real-time tracking. The face feature extraction module is used for extracting phase information from a face region, eliminating interference caused by natural body shaking through a face-body reflection principle, and compensating the phase information through a face symmetry principle to extract effective face motion features. The face key point detection module is used for taking the effective face motion features as neural network input, and detecting the position of face key points by using a space-time fusion network based on a space-time attention mechanism. The millimeter wave radar is arranged in an inclined manner, specifically as follows: 2.The millimeter wave radar-based face key point detection system of claim 1, wherein, (11) The millimeter wave radar is rotated by 90 degrees around the Z-axis of the millimeter wave coordinate system, so that the direction of the millimeter wave antenna array is parallel to the vertical direction to capture the vertical motion of the user's face muscles; (12) The millimeter wave radar is rotated by 20-40 degrees around the Y-axis of the millimeter wave coordinate system, so that it is located in the oblique front of the user, and the millimeter wave propagation direction is oblique to the face of the user, to capture the horizontal motion of the user's face muscles. The user head detection module determines the distance range where the user's face is located on the distance-angle graph through a constant false alarm rate detection algorithm, and the specific steps are as follows: 3.The millimeter wave radar based facial landmark detection system of claim 1, wherein, (21) The false alarm rate p of the constant false alarm rate detection algorithm is set; (22) The energy threshold of each point on the distance-angle graph is calculated based on the false alarm rate p of the constant false alarm rate detection algorithm, and the points with actual energy greater than the energy threshold are taken as detection targets; The user head detection module dynamically adjusts the speed resolution of the millimeter wave radar through a Doppler super-resolution algorithm, and the Doppler super-resolution algorithm is specifically as follows: (23) Statistically detect the distance range [R min ,R max ] of the target; take R min , as the distance range where the face is located.

4. The millimeter wave radar-based facial landmark detection system of claim 1, wherein, The specific steps for determining the angle range where the user's face is located in the super-resolution speed-angle graph by dynamically adjusting the speed resolution of the millimeter wave radar through the Doppler super-resolution algorithm include: (31) Set the mixed frequency after the millimeter wave data type as [N f , N c , N TX , N RX , N s ], wherein N f is the total number of millimeter wave data frames, N c is the number of chirps contained in each millimeter wave data frame, N TX and N RX are the number of transmitting antennas and the number of receiving antennas of the millimeter wave radar respectively, and N s is the number of samples on each chirp; (32) transforming the original millimeter wave data into reorganized millimeter wave data in a method of sliding window, wherein the sliding window length is equal to a Doppler factor N DF , and the step is 1; the Doppler factor N DF is a parameter set for the head positioning algorithm, and the specific operation of the millimeter wave data reorganization is to reorganize [N f , N c , N TX , N RX , N s ] of the original millimeter wave data into [N f ×N c , N TX ×N RX , N s ]. (33) Perform a discrete Fourier transform on the reconstructed millimeter-wave data along the fast time dimension, and then multiply by the distance resolution d. res The distance information of objects in the scene is obtained; the peak detection algorithm is used to detect the peak region of the distance dimension, which corresponds to the distance range between the user's head and the millimeter-wave radar. (34) Take the reorganized millimeter wave data corresponding to the median of the user's head distance range, and do discrete Fourier transform along its slow time dimension and antenna dimension; multiply the slow time dimension by the velocity resolution v res , corresponding to the velocity information; multiply the antenna dimension by the angle resolution θ res , corresponding to the angle information; the final processing result is the super-resolution velocity-angle diagram.

5. The millimeter wave radar-based facial landmark detection system of claim 4, wherein, (42) The millimeter wave data are converted into a super-resolution speed-angle graph through the Doppler super-resolution algorithm; (41) setting a Doppler factor N DF initial value, velocity significance threshold value ε, and maximum value of Doppler factor (43) The targets in the super-resolution speed-angle graph are detected through the constant false alarm rate detection algorithm, and the spatial clustering algorithm is applied based on the density of the noise to cluster the detection targets into clusters, and the angle range where the clusters are located is recorded. ​ (44) according to calculating a velocity significance index VSI, wherein N clu denotes the number of clusters obtained based on density-based spatial clustering with noise application, and denote the maximum and minimum velocity of the i-th cluster, respectively, and denote the maximum and minimum velocity of all clusters, respectively; (45) determining whether the velocity significance VSI is less than a velocity significance threshold value e, and if VSI < e, setting the Doppler factor N DF = N DF x 2, and repeating steps (42) - (44) until VSI > e or (46) The angle range of the cluster obtained by applying a spatial clustering algorithm to the recorded density-based band noise is taken as the angle range in which the user's face is located. 6.The millimeter wave radar based facial landmark detection system of claim 1, wherein, The face-body reflection principle is specifically: (51) Set the correlation coefficient k of the phase influence of the user's natural body sway on the face area reflection signal and the phase influence on the body area reflection signal; (52) extracting phase information of the face region reflection signal and phase information Φ of the body region reflection signal b (t) = τ b + kΔτ bsway (t), where is the initial phase of the face region reflection signal for the i-th range-angle bin, τ b is the initial phase of the body region reflection signal, is the phase influence of the face motion on the face region reflection signal for the i-th range-angle bin, Δτ bsway (t) is the phase influence of the body natural sway on the face region reflection signal; (53) Calculate the angle corresponding to the ith distance-angle bin pair and the angle of the body region corresponding bin Then the correlation coefficient (54) combining the phase information after interference cancellation obtained from step (52) and step (53) 7. The millimeter wave radar-based facial landmark detection system of claim 6, wherein, The face symmetry principle is specifically: (61) First-order difference ΔΦ is taken for the phase information of the face region, and displacement of each face bin is calculated according to the difference result (62) Calculate the angle between the face motion and the face orientation by least squares As follows: wherein, and are the displacements of the left and right face, respectively, of the ith symmetric bin, N sym is the total number of symmetric bins. (63) The RANSAC algorithm is used to judge the outlier data in the calculation process of step (61), and the outlier data is marked as an asymmetric area, and the remaining face bins are marked as symmetric areas; (64) If the millimeter wave radar is deployed on the left side of the face, the phase compensation needed to calculate each bin of the right side face is calculated; if the bin of the right side face belongs to the symmetric region, the first-order difference of the compensated phase where ΔR l is the displacement measured at the bin of the left side face symmetric to the bin of the right side face; if the bin of the right side face belongs to the asymmetric region, the first-order difference of the compensated phase where ΔR r is the displacement measured at the bin.

8. The millimeter wave radar-based facial landmark detection system of claim 7, wherein, The spatiotemporal fusion network structure based on the spatiotemporal attention mechanism consists of the following layers from input to output: a layer normalization layer, a spatiotemporal attention layer, a cascaded layer, another layer normalization layer, and a linear layer. The spatiotemporal attention layer contains two branches, each consisting of a 68-head attention layer connected to a linear layer. The spatiotemporal fusion network based on the spatiotemporal attention mechanism is of type [N]. f ×N c N bin The facial motion features are used as input, N f N represents the total number of millimeter-wave data frames. c N represents the number of chirps contained in each frame of millimeter-wave data. bin The number of face bins is denoted by '1'. After token embedding and position encoding, the face motion features are input into the network, and attention weights are calculated along the two dimensions of the face motion features. 9.A method for face key point detection based on millimeter wave radar, based on the system of any one of claims 1-8, characterized in that, The method includes a calibration stage and a detection stage; Calibration stage: 11) Deploy the millimeter wave radar in the scene in a tilted deployment manner, and detect the user who keeps natural head shaking in the scene; 12) The millimeter wave radar receives the echo signal reflected by the human body, and generates millimeter wave intermediate frequency signal data through mixing processing; 13) Convert the millimeter wave intermediate frequency signal data into a distance-angle graph; determine the distance range in which the face is located on the distance-angle graph; 14) Dynamically adjust the speed resolution of the millimeter wave radar, and determine the angle range in which the face is located in the super-resolution speed-angle graph; 15) Determine the initial position of the face on the distance-angle graph in combination with the distance range and the angle range in which the face is located, and record the historical trajectory information of the face position; Detection stage: 21) Deploy the millimeter wave radar in the scene in a tilted deployment manner, and detect the user who keeps natural head shaking in the scene; 22) The millimeter wave radar receives the echo signal reflected by the human body, and generates millimeter wave intermediate frequency signal data through mixing processing; 23) Convert the millimeter wave intermediate frequency signal data into a distance-angle graph; determine the distance range in which the face is located on the distance-angle graph; 24) Dynamically adjust the speed resolution of the millimeter wave radar, and determine the angle range in which the face is located in the super-resolution speed-angle graph; 25) Determine the position of the face on the distance-angle graph in combination with the distance range and the angle range in which the face is located, as the measured position of the face at the current time; 26) Take the measured position of the face at the current time and the historical trajectory information as the input of the Kalman filter to obtain the final position estimation of the face; then add the measured position of the face at the current time to the historical trajectory information of the face position; 27) Extract the phase information of the region corresponding to the face position; eliminate the interference caused by the natural body sway, compensate the phase information, and extract the effective face motion feature; 28) Take the face motion feature as the input of the neural network, and use the spatio-temporal fusion network based on the spatio-temporal attention mechanism to detect the face key point position.

Images