Millimeter wave radar-based face key point detection system and method

Abstract

The invention discloses a face key point detection system and method based on a millimeter wave radar. The system comprises a millimeter wave data acquisition module, a user head detection module, a face feature extraction module and a face key point detection module. Millimeter wave data are collected through a millimeter wave data collection module, and horizontal movement and vertical movement of face muscles of a user are captured at the same time; determining the position of the face of the user on the distance-angle graph through a user head detection module; interference generated by natural shaking of the body is eliminated through a face feature extraction module, and phase information is compensated to extract effective face motion features; and finally, face key points are detected through a face key point detection module. According to the invention, high-precision face key point detection based on the millimeter-wave radar is realized, and a basic technical support is provided for various man-machine interaction applications such as subsequent expression recognition, fatigue monitoring and augmented reality.

Description

Technical Field

[0001] The present invention belongs to the technical field of millimeter wave wireless sensing and human-computer interaction, and in particular relates to a human face key point detection system and method based on millimeter wave radar. Background Art

[0002] As one of the most common and intuitive facial feature representations, facial key points have been widely used in a variety of human-computer interaction applications such as expression recognition, fatigue monitoring, augmented reality, and virtual reality. These applications rely on high-precision facial key point detection results, otherwise their performance will be significantly reduced. For example, if the facial key point detection system cannot accurately identify the key points of the mouth, the expression recognition system will have difficulty distinguishing the opening and closing of the user's mouth or the movement of the mouth corners, which may lead to erroneous expression recognition results. Therefore, high-precision facial key point systems and methods are crucial to the performance of these applications.

[0003] At present, the main solutions for face key point detection include the following:

[0004] (1) Detection methods based on computer vision: These methods usually use deep neural networks to identify facial key points in images. However, due to the inherent limitations of cameras, these methods have the risk of leaking user privacy, are very sensitive to lighting conditions, and are easily affected by occlusion of facial accessories such as glasses and masks.

[0005] (2) Detection methods based on wearable devices: These methods usually rely on special equipment to read the electromyographic signals of the user's face to infer the location of facial key points. However, due to the limitation of perception accuracy, these methods are difficult to accurately detect facial key points. In addition, they cannot perceive the orientation of the user's head, which further limits their applicability in practical applications.

[0006] (3) Detection method based on millimeter-wave radar: The spatial resolution of the traditional detection method based on millimeter-wave radar is relatively low. Although the method based on synthetic aperture radar algorithm can achieve higher spatial resolution, it faces great challenges in terms of time efficiency.

[0007] (4) Detection methods based on other wireless devices: Using RFID (Radio Frequency Identification) devices, Wi-Fi devices or ultrasonic devices to perceive the facial state. The spatial resolution of these methods is relatively limited and cannot achieve accurate detection of facial key points.

[0008] Therefore, based on the above considerations, it is necessary to propose a facial key point detection system and method based on millimeter-wave radar, which can achieve a balance between spatial resolution and time efficiency, thereby realizing efficient and accurate facial key point detection. Summary of the invention

[0009] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a facial key point detection system and method based on millimeter-wave radar, so as to solve the problems in the existing facial key point detection technology that the camera is easy to leak privacy, is sensitive to lighting conditions and cannot penetrate obstructions, and the problem that most detection methods based on wireless devices usually cannot achieve sufficient resolution, or it is difficult to achieve a balance between spatial resolution and time efficiency.

[0010] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A human face key point detection system based on millimeter wave radar of the present invention comprises: a millimeter wave data acquisition module, a user head detection module, a human face feature extraction module and a human face key point detection module;

[0012] The millimeter wave data acquisition module includes a millimeter wave radar and a data transmission unit;

[0013] Millimeter wave radar is used to transmit wireless radio frequency signals and receive echo signals reflected by the human body, and generate millimeter wave intermediate frequency signal data through mixing processing;

[0014] A data transmission unit, used to send millimeter wave intermediate frequency signal data to a user head detection module;

[0015] The user head detection module is used to convert the millimeter wave intermediate frequency signal data into a distance-angle map through 3D Fourier transform; determine the distance range of the user's face; dynamically adjust the speed resolution of the millimeter wave radar to generate a super-resolution speed-angle map; use the speed difference between the user's head and body to determine the angle range of the user's face in the super-resolution speed-angle map; determine the position of the face on the distance-angle map in combination with the distance range and angle range of the user's face, and perform real-time tracking;

[0016] The facial feature extraction module is used to extract phase information from the facial area, eliminate interference caused by natural body shaking through the face-body reflection principle, and then compensate for the phase information through the face symmetry principle to extract effective facial motion features;

[0017] The facial key point detection module is used to use effective facial motion features as neural network input and detect the positions of facial key points using a spatial-temporal fusion network (STFN) based on a spatiotemporal attention mechanism.

[0018] Furthermore, the millimeter-wave radar is arranged in an inclined manner, so that the millimeter-wave radar can have a higher spatial resolution.

[0019] Furthermore, the millimeter wave radar tilt deployment setting is specifically as follows:

[0020] (11) Rotate the millimeter wave radar 90° around the Z axis of the millimeter wave coordinate system so that the direction of the millimeter wave antenna array is parallel to the vertical direction to capture the vertical movement of the user's facial muscles;

[0021] (12) The millimeter wave radar is rotated 20° to 40° around the Y axis of the millimeter wave coordinate system so that it is located diagonally in front of the user and the millimeter wave propagation direction is diagonally toward the user's face to capture the horizontal movement of the user's facial muscles.

[0022] Furthermore, the user head detection module determines the distance range of the user's face on the distance-angle graph by a constant false alarm rate (CFAR) detection algorithm, and the specific steps are as follows:

[0023] (21) Setting the false alarm rate of the constant false alarm rate detection algorithm to p;

[0024] (22) calculating the energy threshold of each point on the distance-angle graph based on the false alarm rate p of the constant false alarm rate detection algorithm, and taking the point whose actual energy is greater than the energy threshold as the detection target;

[0025] (23) Statistical detection target distance range [R min ,R max ]; R min , The distance range where the face is located.

[0026] Furthermore, the user head detection module dynamically adjusts the velocity resolution of the millimeter wave radar through a Doppler super-resolution algorithm. The Doppler super-resolution algorithm is specifically as follows:

[0027] (31) Set the millimeter wave data type after mixing to [N f ,N c ,NT X ,N RX ,N s ], where N f is the total number of millimeter wave data frames, N c N is the number of chirps contained in each frame of millimeter wave data. TX and N RX are the number of transmitting antennas and receiving antennas of millimeter wave radar, N s is the number of samples for each chirp;

[0028] (32) The original millimeter wave data is transformed into reconstructed millimeter wave data using a sliding window method, where the sliding window length is equal to the Doppler factor N DF , step size is 1; Doppler factor N DF is the parameter set for the head positioning algorithm. The specific operation of millimeter wave data reorganization is to convert the original millimeter wave data [N f ,N c ,N TX ,N RX ,N s ] is reorganized into [N f ×N c ,N TX ×N RX ,N s ];

[0029] (33) The reconstructed millimeter wave data is discretely Fourier transformed along the fast time dimension (i.e., the last dimension) and then multiplied by the distance resolution d res , obtain the distance information of the objects in the scene; use the peak detection algorithm to detect the peak area of ​​the distance dimension, which corresponds to the distance range between the user's head and the millimeter wave radar;

[0030] (34) Take the reconstructed millimeter wave data corresponding to the median of the user head distance range, and perform discrete Fourier transform along its slow time dimension (i.e., the first dimension) and antenna dimension (i.e., the second dimension); multiply the slow time dimension by the velocity resolution v res , corresponding to the velocity information; the antenna dimension multiplied by the angular resolution θ res , corresponding to the angle information; the final processing result is a super-resolution speed-angle map.

[0031] Furthermore, the specific steps of dynamically adjusting the velocity resolution of the millimeter wave radar by using the Doppler super-resolution algorithm and determining the angle range of the user's face in the super-resolution velocity-angle map include:

[0032] (41) Set the Doppler factor N DF Initial value, velocity significance threshold ∈ and Doppler factor maximum value

[0033] (42) Converting millimeter wave data into super-resolution velocity-angle images by Doppler super-resolution algorithm;

[0034] (43) Detecting targets in the super-resolution speed-angle image using a constant false alarm rate detection algorithm; clustering the detected targets into clusters using a density-based spatial clustering of applications with noise (DBSCAN) algorithm; and recording the angle ranges of these clusters;

[0035] (44) According to Calculate the velocity significance index VSI, where N clu represents the number of clusters obtained by applying density-based spatial clustering with noise, and Respectively represent the maximum speed and minimum speed (can be negative) of the i-th cluster, and Respectively represent the maximum speed and minimum speed of all clusters (can be negative);

[0036] (45) Determine whether the velocity significance VSI is less than the velocity significance threshold ∈. If VSI<∈, then set the Doppler factor N DF =N DF ×2, repeat steps (42) to (44) until VSI ≥ ∈ or

[0037] (46) The angle range of the cluster obtained by applying the density-based noise spatial clustering algorithm is recorded as the angle range of the user's face.

[0038] Furthermore, the face-body reflection principle is specifically as follows:

[0039] (51) setting a correlation coefficient k between the influence of the natural shaking of the user's body on the phase of the reflected signal in the face area and the phase of the reflected signal in the body area;

[0040] (52) Extract the phase information of the reflected signal from the face area and the phase information Φ of the reflected signal of the body area b (t) = τ b +kΔτ bsway (t), where is the initial phase of the reflected signal of the face area in the i-th distance-angle bin, τ b is the initial phase of the reflected signal from the body area, is the phase effect of face motion on the reflected signal of the face area in the i-th distance-angle bin, Δτ bsway (t) is the phase effect of the natural shaking of the body on the reflected signal of the face area;

[0041] (53) Calculate the angle corresponding to the i-th distance-angle bin And the angle of the body area corresponding to the bin The correlation coefficient

[0042] (54) Combine steps (52) and (53) to obtain the phase information after interference elimination

[0043] Furthermore, the face symmetry principle is specifically:

[0044] (61) Calculate the first-order difference ΔΦ of the phase information of the face area, and calculate the displacement of each face bin based on the difference result

[0045] (62) Calculate the angle between facial movement and face orientation using the least squares method as follows:

[0046]

[0047] in, and are the displacements of the i-th symmetric bins of the left and right faces, respectively, N sym is the total number of symmetric bins;

[0048] (63) Using the RANSAC algorithm to determine the outlier data in the calculation process of step (61), mark it as an asymmetric area, and mark the remaining face bins as symmetric areas;

[0049] (64) Assuming that the millimeter-wave radar is deployed on the left side of the face, the phase compensation required for each bin located on the right side of the face is calculated; if the bin located on the right side of the face belongs to the symmetrical area, the first-order difference of the phase after compensation is Among them, ΔR l is the displacement of the bin on the left face that is symmetrical to the bin on the right face. If the bin on the right face belongs to an asymmetric area, the first-order difference of the phase after compensation is Among them, ΔR r is the measured displacement at the bin.

[0050] Furthermore, the network structure of the spatiotemporal fusion network based on the spatiotemporal attention mechanism is as follows from the input end to the output end: layer normalization layer, spatiotemporal attention layer, cascade layer, layer normalization layer and linear layer; wherein the spatiotemporal attention layer contains two branches, each of which is a 68-head attention layer connected to a linear layer; the spatiotemporal fusion network based on the spatiotemporal attention mechanism is of type [N f ×N c ,N bin ]’s facial motion features as input, N f is the total number of millimeter wave data frames, N c is the number of chirps contained in each frame of millimeter wave data, N binis the number of face bins; the facial motion features are token embedded and position encoded, and then input into the network to calculate the attention weights along the two dimensions of the facial motion features.

[0051] A method for detecting key points of human faces based on millimeter wave radar of the present invention is based on the above system and includes a calibration phase and a detection phase;

[0052] Calibration phase:

[0053] 11) Deploy the millimeter-wave radar in the scene in an inclined deployment mode and detect users who keep shaking their heads naturally in the scene;

[0054] 12) The millimeter wave radar receives the echo signal reflected by the human body and generates millimeter wave intermediate frequency signal data through mixing processing;

[0055] 13) Converting the millimeter wave intermediate frequency signal data into a distance-angle graph through 3D Fourier transform; determining the distance range of the face on the distance-angle graph;

[0056] 14) Dynamically adjust the velocity resolution of the millimeter wave radar to determine the angle range of the face in the super-resolution velocity-angle map;

[0057] 15) Determine the initial position of the face on the distance-angle graph based on the distance range and angle range of the face, and record it in the historical trajectory information of the face position;

[0058] Detection phase:

[0059] 21) Deploy the millimeter wave radar in the scene in an inclined deployment mode and detect users moving freely in the scene;

[0060] 22) The millimeter wave radar receives the echo signal reflected by the human body and generates millimeter wave intermediate frequency signal data through mixing processing;

[0061] 23) Converting the millimeter wave intermediate frequency signal data into a distance-angle graph by 3D Fourier transform; determining the distance range of the face on the distance-angle graph;

[0062] 24) Dynamically adjust the velocity resolution of the millimeter wave radar to determine the angle range of the face in the super-resolution velocity-angle map;

[0063] 25) Determine the position of the face on the distance-angle graph based on the distance range and angle range of the face, as the measurement position of the face at the current moment;

[0064] 26) Using the current face measurement position and historical trajectory information as inputs to the Kalman filter to obtain a final face position estimate; then adding the current face measurement position to the historical trajectory information of the face position;

[0065] 27) Extracting the phase information of the area corresponding to the face position; eliminating the interference caused by the natural shaking of the body, compensating the phase information, and extracting effective face motion features;

[0066] 28) The facial motion features are used as the neural network input, and the locations of facial key points are detected using the spatiotemporal fusion network based on the spatiotemporal attention mechanism.

[0067] Beneficial effects of the present invention:

[0068] 1. The present invention can capture the horizontal and vertical movements of the user's facial muscles respectively, thereby improving the integrity of facial motion feature extraction and obtaining a higher spatial resolution without using a synthetic aperture radar algorithm.

[0069] 2. The present invention can calculate the position of a face on a distance-angle diagram, and then spatially distinguish the face reflection signal from the body reflection signal, thereby avoiding interference caused by the aliasing of the face reflection signal and the body reflection signal.

[0070] 3. The present invention eliminates the interference caused by the natural shaking of the body, and performs phase compensation on the reflected signal with weaker energy during the detection process, thereby enhancing the signal-to-noise ratio of the facial motion features and improving the accuracy of the facial key point detection results.

[0071] 4. The present invention realizes an end-to-end facial key point detection neural network based on millimeter wave data, and the average error of facial key points is only 2.81mm.

[0072] 5. The present invention realizes high-precision detection of facial key points based on millimeter-wave radar, providing basic technical support for subsequent expression recognition, fatigue monitoring, augmented reality and other human-computer interaction applications; the present invention also has low environmental requirements, is not easily disturbed, has high robustness, and can work normally in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a structural diagram of the facial key point detection system of the present invention;

[0074] Figure 2 A schematic diagram of the tilted deployment of the millimeter wave radar of the present invention;

[0075] Figure 3a is a schematic diagram of a distance-angle graph after 3D Fourier transformation of the present invention;

[0076] Figure 3b A schematic diagram of a super-resolution velocity-angle diagram generated by the Doppler super-resolution algorithm of the present invention;

[0077] Figure 4 It is a schematic diagram of the face-body reflection principle of the present invention;

[0078] Figure 5 It is a schematic diagram of the principle of face symmetry of the present invention;

[0079] Figure 6 A schematic diagram of the network structure of the spatiotemporal fusion network of the present invention;

[0080] Figure 7 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0081] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0082] Reference Figures 1 to 6 As shown, a face key point detection system based on millimeter wave radar of the present invention comprises: a millimeter wave data acquisition module, a user head detection module, a face feature extraction module and a face key point detection module;

[0083] The millimeter wave data acquisition module includes a millimeter wave radar and a data transmission unit;

[0084] Millimeter wave radar is used to transmit wireless radio frequency signals and receive echo signals reflected by the human body, and generate millimeter wave intermediate frequency signal data through mixing processing;

[0085] A data transmission unit, used to send millimeter wave intermediate frequency signal data to a user head detection module;

[0086] Specifically, the millimeter wave radar is arranged in an inclined manner so that the millimeter wave radar can have a higher spatial resolution;

[0087] The millimeter wave radar tilt deployment setting is specifically as follows:

[0088] (11) Rotate the millimeter wave radar 90° around the Z axis of the millimeter wave coordinate system so that the direction of the millimeter wave antenna array is parallel to the vertical direction to capture the vertical movement of the user's facial muscles;

[0089] (12) The millimeter wave radar is rotated 20° to 40° around the Y axis of the millimeter wave coordinate system so that it is located diagonally in front of the user and the millimeter wave propagation direction is diagonally toward the user's face to capture the horizontal movement of the user's facial muscles.

[0090] The user head detection module is used to convert the millimeter wave intermediate frequency signal data into a distance-angle diagram through 3D Fourier transform; determine the distance range of the user's face on the distance-angle diagram through a constant false alarm rate (CFAR) detection algorithm; dynamically adjust the speed resolution of the millimeter wave radar through a Doppler super-resolution algorithm to generate a super-resolution speed-angle diagram; use the speed difference between the user's head and body to determine the angle range of the user's face in the super-resolution speed-angle diagram; determine the position of the user's face on the distance-angle diagram in combination with the distance range and angle range of the user's face, and perform real-time tracking;

[0091] Specifically, the user head detection module determines the distance range of the user's face on the distance-angle graph through a constant false alarm rate (CFAR) detection algorithm, and the specific steps are as follows:

[0092] (21) Setting the false alarm rate of the constant false alarm rate detection algorithm to p;

[0093] (22) calculating the energy threshold of each point on the distance-angle graph based on the false alarm rate p of the constant false alarm rate detection algorithm, and taking the point whose actual energy is greater than the energy threshold as the detection target;

[0094] (23) Statistical detection target distance range [R min ,R max ]; R min , The distance range where the face is located.

[0095] Wherein, the Doppler super-resolution algorithm is specifically as follows:

[0096] (31) Set the millimeter wave data type after mixing to [N f ,N c ,N TX ,N RX ,N s ], where N f is the total number of millimeter wave data frames, N c N is the number of chirps contained in each frame of millimeter wave data. TX and N RX are the number of transmitting antennas and receiving antennas of millimeter wave radar, N s is the number of samples for each chirp;

[0097] (32) The original millimeter wave data is transformed into reconstructed millimeter wave data using a sliding window method, where the sliding window length is equal to the Doppler factor N DF , step size is 1; Doppler factor N DFis the parameter set for the head positioning algorithm. The specific operation of millimeter wave data reorganization is to convert the original millimeter wave data [N f ,N c ,N TX ,N RX ,N s ] is reorganized into [N f ×N c ,N TX ×N RX ,N s ];

[0098] (33) The reconstructed millimeter wave data is discretely Fourier transformed along the fast time dimension (i.e., the last dimension) and then multiplied by the distance resolution d res , obtain the distance information of the objects in the scene; use the peak detection algorithm to detect the peak area of ​​the distance dimension, which corresponds to the distance range between the user's head and the millimeter wave radar;

[0099] (34) Take the reconstructed millimeter wave data corresponding to the median of the user head distance range, and perform discrete Fourier transform along its slow time dimension (i.e., the first dimension) and antenna dimension (i.e., the second dimension); multiply the slow time dimension by the velocity resolution v res , corresponding to the velocity information; the antenna dimension multiplied by the angular resolution θ res , corresponding to the angle information; the final processing result is a super-resolution speed-angle map.

[0100] The specific steps of dynamically adjusting the velocity resolution of the millimeter wave radar by using the Doppler super-resolution algorithm and determining the angle range of the user's face in the super-resolution velocity-angle map include:

[0101] (41) Set the Doppler factor N DF Initial value, velocity significance threshold ∈ and Doppler factor maximum value

[0102] (42) Converting millimeter wave data into super-resolution velocity-angle images by Doppler super-resolution algorithm;

[0103] (43) Detecting targets in the super-resolution speed-angle image using a constant false alarm rate detection algorithm; clustering the detected targets into clusters using a density-based spatial clustering of applications with noise (DBSCAN) algorithm; and recording the angle ranges of these clusters;

[0104] (44) According to Calculate the velocity significance index VSI, where N clurepresents the number of clusters obtained by applying density-based spatial clustering with noise, and Respectively represent the maximum speed and minimum speed (can be negative) of the i-th cluster, and Respectively represent the maximum speed and minimum speed of all clusters (can be negative);

[0105] (45) Determine whether the velocity significance VSI is less than the velocity significance threshold ∈. If VSI<∈, then set the Doppler factor N DF =N DF ×2, repeat steps (42) to (44) until VSI ≥ ∈ or

[0106] (46) The angle range of the cluster obtained by applying the density-based noise spatial clustering algorithm is recorded as the angle range of the user's face.

[0107] The facial feature extraction module is used to extract phase information from the facial area, eliminate interference caused by natural body shaking through the face-body reflection principle, and then compensate for the phase information through the face symmetry principle to extract effective facial motion features;

[0108] The face-body reflection principle is as follows:

[0109] (51) setting a correlation coefficient k between the influence of the natural shaking of the user's body on the phase of the reflected signal in the face area and the phase of the reflected signal in the body area;

[0110] (52) Extract the phase information of the reflected signal from the face area and the phase information Φ of the reflected signal of the body area b (t) = τ b +kΔτ bsway (t), where is the initial phase of the reflected signal of the face area in the i-th distance-angle bin, τ b is the initial phase of the reflected signal from the body area, is the phase effect of face motion on the reflected signal of the face area in the i-th distance-angle bin, Δτ bsway (t) is the phase effect of the natural shaking of the body on the reflected signal of the face area;

[0111] (53) Calculate the angle corresponding to the i-th distance-angle bin And the angle of the body area corresponding to the bin The correlation coefficient

[0112] (54) Combine steps (52) and (53) to obtain the phase information after interference elimination

[0113] The face symmetry principle is specifically:

[0114] (61) Calculate the first-order difference ΔΦ of the phase information of the face area, and calculate the displacement of each face bin based on the difference result

[0115] (62) Calculate the angle between facial movement and face orientation using the least squares method as follows:

[0116]

[0117] in, and are the displacements of the i-th symmetric bins of the left and right faces, respectively, N sym is the total number of symmetric bins;

[0118] (63) Using the RANSAC algorithm to determine the outlier data in the calculation process of step (61), mark it as an asymmetric area, and mark the remaining face bins as symmetric areas;

[0119] (64) Assuming that the millimeter-wave radar is deployed on the left side of the face, the phase compensation required for each bin located on the right side of the face is calculated; if the bin located on the right side of the face belongs to the symmetrical area, the first-order difference of the phase after compensation is Among them, ΔR l is the displacement of the bin on the left face that is symmetrical to the bin on the right face. If the bin on the right face belongs to an asymmetric area, the first-order difference of the phase after compensation is Among them, ΔR r is the measured displacement at the bin.

[0120] The facial key point detection module is used to use effective facial motion features as neural network input and detect the positions of facial key points using a spatial-temporal fusion network (STFN) based on a spatiotemporal attention mechanism.

[0121] Specifically, the network structure of the spatiotemporal fusion network based on the spatiotemporal attention mechanism is as follows from the input end to the output end: layer normalization layer, spatiotemporal attention layer, cascade layer, layer normalization layer and linear layer; wherein the spatiotemporal attention layer contains two branches, each of which is a 68-head attention layer connected to a linear layer; the spatiotemporal fusion network based on the spatiotemporal attention mechanism is of type [N f ×N c ,N bin ]’s facial motion features as input, N f is the total number of millimeter wave data frames, N c is the number of chirps contained in each frame of millimeter wave data, N bin is the number of face bins; the facial motion features are token embedded and position encoded, and then input into the network to calculate the attention weights along the two dimensions of the facial motion features.

[0122] Reference Figure 7 As shown, a method for detecting key points of human faces based on millimeter wave radar of the present invention is based on the above system and includes a calibration phase and a detection phase;

[0123] Calibration phase:

[0124] 11) Deploy the millimeter-wave radar in the scene in an inclined deployment mode and detect users who keep shaking their heads naturally in the scene;

[0125] 12) The millimeter wave radar receives the echo signal reflected by the human body and generates millimeter wave intermediate frequency signal data through mixing processing;

[0126] 13) Converting the millimeter wave intermediate frequency signal data into a distance-angle diagram through 3D Fourier transform; determining the distance range of the face on the distance-angle diagram through a constant false alarm rate detection algorithm;

[0127] 14) Dynamically adjust the velocity resolution of the millimeter-wave radar through the Doppler super-resolution algorithm, and determine the angle range of the face in the super-resolution velocity-angle map;

[0128] 15) Determine the initial position of the face on the distance-angle graph based on the distance range and angle range of the face, and record it in the historical trajectory information of the face position.

[0129] Detection phase:

[0130] 21) Deploy the millimeter wave radar in the scene in an inclined deployment mode and detect users moving freely in the scene;

[0131] 22) The millimeter wave radar receives the echo signal reflected by the human body and generates millimeter wave intermediate frequency signal data through mixing processing;

[0132] 23) Converting the millimeter wave intermediate frequency signal data into a distance-angle diagram through 3D Fourier transform; determining the distance range of the face on the distance-angle diagram through a constant false alarm rate detection algorithm;

[0133] 24) Dynamically adjust the velocity resolution of the millimeter wave radar through the Doppler super-resolution algorithm, and determine the angle range of the face in the super-resolution velocity-angle map;

[0134] 25) Determine the position of the face on the distance-angle graph based on the distance range and angle range of the face, as the measurement position of the face at the current moment;

[0135] 26) Using the current face measurement position and historical trajectory information as inputs to the Kalman filter to obtain a final face position estimate; then adding the current face measurement position to the historical trajectory information of the face position;

[0136] 27) Extracting the phase information of the area corresponding to the face position; eliminating the interference caused by the natural shaking of the body through the face-body reflection principle, and then compensating the phase information through the face symmetry principle to extract effective face motion features;

[0137] 28) The facial motion features are used as the neural network input, and the locations of facial key points are detected using the spatiotemporal fusion network based on the spatiotemporal attention mechanism.

[0138] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A facial key point detection system based on millimeter wave radar, characterized in that: include: Millimeter wave data acquisition module, user head detection module, face feature extraction module and face key point detection module; The millimeter wave data acquisition module includes a millimeter wave radar and a data transmission unit; Millimeter wave radar is used to transmit wireless radio frequency signals and receive echo signals reflected by the human body, and generate millimeter wave intermediate frequency signal data through mixing processing; A data transmission unit, used to send millimeter wave intermediate frequency signal data to a user head detection module; The user head detection module is used to convert the millimeter wave intermediate frequency signal data into a distance-angle map through 3D Fourier transform; determine the distance range of the user's face; dynamically adjust the speed resolution of the millimeter wave radar to generate a super-resolution speed-angle map; use the speed difference between the user's head and body to determine the angle range of the user's face in the super-resolution speed-angle map; determine the position of the face on the distance-angle map in combination with the distance range and angle range of the user's face, and perform real-time tracking; The facial feature extraction module is used to extract phase information from the facial area, eliminate interference caused by natural body shaking through the face-body reflection principle, and then compensate for the phase information through the face symmetry principle to extract effective facial motion features; The facial key point detection module is used to use effective facial motion features as neural network input and detect the positions of facial key points using a spatiotemporal fusion network based on a spatiotemporal attention mechanism.

2. The face key point detection system based on millimeter wave radar according to claim 1 is characterized in that: The millimeter wave radar tilt deployment setting is specifically as follows: (11) Rotate the millimeter wave radar 90° around the Z axis of the millimeter wave coordinate system so that the direction of the millimeter wave antenna array is parallel to the vertical direction to capture the vertical movement of the user's facial muscles; (12) The millimeter wave radar is rotated 20° to 40° around the Y axis of the millimeter wave coordinate system so that it is located diagonally in front of the user and the millimeter wave propagation direction is diagonally toward the user's face to capture the horizontal movement of the user's facial muscles.

3. The face key point detection system based on millimeter wave radar according to claim 1 is characterized in that: The user head detection module determines the distance range where the user's face is located on the distance-angle graph by using a constant false alarm rate detection algorithm. The specific steps are as follows: (21) Setting the false alarm rate of the constant false alarm rate detection algorithm to p; (22) calculating the energy threshold of each point on the distance-angle graph based on the false alarm rate p of the constant false alarm rate detection algorithm, and taking the point whose actual energy is greater than the energy threshold as the detection target; (23) Statistical detection target distance range [R min ,R max ]; R min , The distance range where the face is located.

4. The face key point detection system based on millimeter wave radar according to claim 1 is characterized in that: The user head detection module dynamically adjusts the velocity resolution of the millimeter wave radar through the Doppler super-resolution algorithm. The Doppler super-resolution algorithm is as follows: (31) Set the millimeter wave data type after mixing to [N f , N c , N TX , N RX , N s ], where N f is the total number of millimeter wave data frames, N c is the number of chirps contained in each frame of millimeter wave data, N TX and N RX are the number of transmitting antennas and receiving antennas of millimeter wave radar, N s is the number of samples for each chirp; (32) The original millimeter wave data is transformed into reconstructed millimeter wave data using a sliding window method, where the sliding window length is equal to the Doppler factor N DF , step size is 1; Doppler factor N DF is the parameter set for the head positioning algorithm. The specific operation of millimeter wave data reorganization is to convert the original millimeter wave data [N f , N c , N TX , N RX , N s ] is reorganized into [N f ×N c , N TX ×N RX , N s ]; (33) The reconstructed millimeter wave data is discretely Fourier transformed along the fast time dimension and then multiplied by the distance resolution d res , obtain the distance information of the objects in the scene; use the peak detection algorithm to detect the peak area of ​​the distance dimension, which corresponds to the distance range between the user's head and the millimeter wave radar; (34) Take the reconstructed millimeter wave data corresponding to the median of the user head distance range, perform discrete Fourier transform along its slow time dimension and antenna dimension; multiply the slow time dimension by the velocity resolution v res , corresponding to the velocity information; the antenna dimension multiplied by the angular resolution θ res , corresponding to the angle information; the final processing result is a super-resolution speed-angle map.

5. The face key point detection system based on millimeter wave radar according to claim 4 is characterized in that: The specific steps of dynamically adjusting the velocity resolution of the millimeter wave radar by using the Doppler super-resolution algorithm and determining the angle range of the user's face in the super-resolution velocity-angle map include: (41) Set the Doppler factor N DF Initial value, velocity significance threshold ∈ and Doppler factor maximum value (42) Converting millimeter wave data into super-resolution velocity-angle images by Doppler super-resolution algorithm; (43) Detecting targets in the super-resolution speed-angle image using a constant false alarm rate detection algorithm, and clustering the detected targets into clusters using a density-based noise-band spatial clustering algorithm, and recording the angle ranges of these clusters; (44) According to Calculate the velocity significance index VSI, where N clu represents the number of clusters obtained by applying density-based spatial clustering with noise, and Respectively represent the maximum speed and minimum speed of the ith cluster, and Respectively represent the maximum and minimum speeds of all clusters; (45) Determine whether the velocity significance VSI is less than the velocity significance threshold ∈. If VSI < ∈, then set the Doppler factor N DF =N DF ×2, repeat steps (42) to (44) until VSI ≥ ∈ or (46) The angle range of the cluster obtained by applying the density-based noise spatial clustering algorithm is recorded as the angle range of the user's face.

6. The face key point detection system based on millimeter wave radar according to claim 1, characterized in that: The principle based on face-body reflection is specifically: (51) setting a correlation coefficient k between the influence of the natural shaking of the user's body on the phase of the reflected signal in the face area and the phase of the reflected signal in the body area; (52) Extract the phase information of the reflected signal from the face area and the phase information Φ of the reflected signal of the body area b (t) = τ b +kΔτ bsway (t), where is the initial phase of the reflected signal of the face area in the i-th distance-angle bin, τ b is the initial phase of the reflected signal from the body area, is the phase effect of face motion on the reflected signal of the face area in the i-th distance-angle bin, Δτ bsway (t) is the phase effect of the natural shaking of the body on the reflected signal of the face area; (53) Calculate the angle corresponding to the i-th distance-angle bin And the angle of the body area corresponding to the bin The correlation coefficient (54) Combine steps (52) and (53) to obtain the phase information after interference elimination 7. The face key point detection system based on millimeter wave radar according to claim 6 is characterized in that: The face symmetry principle is specifically: (61) Calculate the first-order difference ΔΦ of the phase information of the face area, and calculate the displacement of each face bin based on the difference result (62) Calculate the angle between facial movement and face orientation using the least squares method as follows: in, and are the displacements of the i-th symmetric bins of the left and right faces, respectively, N sym is the total number of symmetric bins; (63) Using the RANSAC algorithm to determine the outlier data in the calculation process of step (61), mark it as an asymmetric area, and mark the remaining face bins as symmetric areas; (64) Assuming that the millimeter-wave radar is deployed on the left side of the face, the phase compensation required for each bin located on the right side of the face is calculated; if the bin located on the right side of the face belongs to the symmetrical area, the first-order difference of the phase after compensation is Among them, ΔR l is the displacement of the bin on the left face that is symmetrical to the bin on the right face. If the bin on the right face belongs to an asymmetric area, the first-order difference of the phase after compensation is Among them, ΔR r is the measured displacement at the bin.

8. The face key point detection system based on millimeter wave radar according to claim 7, characterized in that: The network structure of the spatiotemporal fusion network based on the spatiotemporal attention mechanism is as follows from the input end to the output end: layer normalization layer, spatiotemporal attention layer, cascade layer, layer normalization layer and linear layer; wherein the spatiotemporal attention layer contains two branches, each of which is a 68-head attention layer connected to a linear layer; the spatiotemporal fusion network based on the spatiotemporal attention mechanism is of type [N f ×N c ,N bin ]’s facial motion features as input, N f is the total number of millimeter wave data frames, N c is the number of chirps contained in each frame of millimeter wave data, N bin is the number of face bins; the facial motion features are token embedded and position encoded, and then input into the network to calculate the attention weights along the two dimensions of the facial motion features.

9. A method for detecting key points of a human face based on millimeter wave radar, based on the system described in any one of claims 1 to 8, characterized in that: The method includes a calibration phase and a detection phase; Calibration phase: 11) Deploy the millimeter-wave radar in the scene in an inclined deployment mode and detect users who keep shaking their heads naturally in the scene; 12) The millimeter wave radar receives the echo signal reflected by the human body and generates millimeter wave intermediate frequency signal data through mixing processing; 13) Converting the millimeter wave intermediate frequency signal data into a distance-angle graph; determining the distance range of the face on the distance-angle graph; 14) Dynamically adjust the velocity resolution of the millimeter wave radar to determine the angle range of the face in the super-resolution velocity-angle map; 15) Determine the initial position of the face on the distance-angle graph based on the distance range and angle range of the face, and record it in the historical trajectory information of the face position; Detection phase: 21) Deploy the millimeter wave radar in the scene in an inclined deployment mode and detect users moving freely in the scene; 22) The millimeter wave radar receives the echo signal reflected by the human body and generates millimeter wave intermediate frequency signal data through mixing processing; 23) Converting the millimeter wave intermediate frequency signal data into a distance-angle graph; determining the distance range of the face on the distance-angle graph; 24) Dynamically adjust the velocity resolution of the millimeter wave radar to determine the angle range of the face in the super-resolution velocity-angle map; 25) Determine the position of the face on the distance-angle graph based on the distance range and angle range of the face, as the measurement position of the face at the current moment; 26) Using the current face measurement position and historical trajectory information as inputs to the Kalman filter to obtain a final face position estimate; then adding the current face measurement position to the historical trajectory information of the face position; 27) Extracting the phase information of the area corresponding to the face position; eliminating the interference caused by the natural shaking of the body, compensating the phase information, and extracting effective face motion features; 28) The facial motion features are used as the neural network input, and the locations of facial key points are detected using the spatiotemporal fusion network based on the spatiotemporal attention mechanism.

Images