Facial expression sensing

Abstract

A facial expression sensing method including acquiring radar data from a radar sensor configured to perform range sensing in an environment including a person, analysing the radar data to determine facial radar data indicative of radar data captured from a face of the person, determining a facial expression by using facial radar data and at least one computational model, the at least one computational model being at least partially indicative of relationships between facial radar data and different facial expressions and generating an indication of the facial expression.

Description

FACIAL EXPRESSION SENSING Background of the Invention

[0001] The present invention relates to a system and method for facial expression sensing, and in one particular example, to a system for facial expression sensing using a radar sensor, such as Millimeter wave (mmWave) radar. Description of the Prior Art

[0002] The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgement or admission or any form of suggestion that the prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

[0003] Multimedia Internet of Things (IoT) applications have become popular in recent years, ranging from digital entertainment to digital advertising. To enable better service quality and user experience, emotional awareness has been advocated as a key factor in perception to "understand" audience. Facial expression recognition (FER) plays a vital role in emotional awareness since facial expressions are intuitive reflections of user’s emotional states. A FER system infers facial expressions and delivers an assessment of audience’s preference, interest level, engagement and reactions.

[0004] FER has been extensively studied over the last decade. Vision-based approaches achieve state-of-the-art accuracy, but their performance may be vulnerable to ambient lighting conditions, e.g., watching movies with poor ambient light. The time-of-flight (ToF) camera (e.g., depth camera) may work better in low lighting, but it still suffers from natural illumination (e.g., glare from glasses or exposed to sunlight) and fails with occlusion (e.g., wearing masks). Most importantly, although security measurement can be put in place, cameras may raise serious privacy concerns (e.g., from psychology aspect, people do not feel safe and comfortable with a camera constantly monitoring them).

[0005] Without compromising user privacy, wearable-based approaches have been advocated to recognize facial expressions using wearable sensors, e.g., Photoplethysmography (PPG),electroencephalography (EEG), and earphones, but long-time wearing may cause discomfort to users.

[0006] Device-free approaches have been proposed leveraging on Wi-Fi or ultrasound signals reflected from facial muscle movements. However, due to intrinsic limitations of Wi-Fi signals (i.e., low bandwidth and multipath effect), the WiFi-based approach, described for example in Yanjiao Chen, Runmin Ou, Zhiyang Li, and Kaishun Wu. 2020. WiFace: facial expression recognition using Wi-Fi signals. IEEE Transactions on Mobile Computing 21, 1 (2020), 378– 391, may fail in presence of body motions (e.g., leg shaking).

[0007] The ultrasound-based approach, described for example in Yang Gao, Yincheng Jin, Seokmin Choi, Jiyang Li, Junjie Pan, Lin Shu, Chi Zhou, and Zhanpeng Jin.2021. SonicFace: Tracking Facial Expressions Using a Commodity Microphone Array. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1–33, can reach 60cm at maximum but the distance of watching TV is typically in a range between 1m and 3m. Moreover, these device-free FER approaches typically do not support multi-user applications.

[0008] US20220283296 discloses systems and techniques for detecting user presence, user motion, and for performing facial authentication. For instance, a wireless device can receive a waveform that is a reflection of a transmitted radio frequency (RF) waveform. Based on RF sensing data associated with the received waveform, the wireless device can determine a presence of a user. In response to determining the presence of the user, the wireless device can initiate facial authentication of the user.

[0009] This therefore relates to facial authentication using RF sensing and is not directed to facial expression recognition.

[0010] US20220101650 discloses an electronic device may comprise: an antenna module; a communication circuit; a processor; and a memory, wherein the memory stores instructions which, when executed, cause the processor to: output first beams each having a first beam width to a first spatial range around the electronic device; receive a first reflective pattern with respect to the first beams; determine at least one section in which the external object is disposed, among a plurality of sections configuring the first spatial range; output second beams each having a second beam width to the at least one section; receive a second reflective patternwith respect to the second beams; recognize the external object on the basis of the second reflective pattern of the second beams to authenticate a user; and output third beams to determine state information or motion information of the external object.

[0011] This document uses an mmWave radar to first identify users by their faces, and then it mainly detects motion patterns in mmWave signals, such as large scale gestures, and does not perform facial expression recognition.

[0012] US20220327863 discloses facial identification of subjects using a millimeter wave radar sensor, including: transmitting a radar signal onto faces of subjects using the millimeter wave radar sensor; receiving and accumulating reflected radar data, wherein the reflected radar data is received from channels of multiple receiving antennas; converting the reflected radar data into an image form suitable as an input to a convolutional neural network; and performing the facial identification on the image form using a classifier trained by the convolutional neural network.

[0013] This document therefore proposes an mmWave-based facial authentication system using a trained convolutional neural network and does not perform expression recognition.

[0014] US20230011635 provides a method of facial expression recognition including 3 steps: step 1: collecting facial expression data, which contributes to solve the problem of lacking data, disparate and bias data, that cause the overfitting problem when training the deep learning model; step 2: designing a new deep learning network that able to focus on special regions of the face to extract and learn the important features of facial expressions by intergating ensemble attention modules into basic deep network architecture like ResNet; step 3: training the ensemble attention deep learning model in step 2 on the collected dataset in step 1, using the combination of two loss functions including ArcFace and Softmax to reduce the overfitting problem.

[0015] CN112381061 relates to a facial expression recognition method and a system, comprising the following steps: acquiring a face video, detecting whether a face area exists in each frame of video, if so, entering the next step, and if not, skipping and entering the processing of the next frame of image; detecting a preset number of human face characteristic points in a human face area; calculating and cutting a face region image based on the coordinatepositions of the face feature points; and zooming the cut facial region image, inputting the trained facial expression recognition network model embedded in the lightweight space attention module, obtaining the prediction probability value of each emotion category, and outputting the category with the maximum probability as an expression recognition result. The invention can identify 7 types of basic expressions and 1 neutral expression of the human face, and can improve the identification precision of the facial expression in an uncontrolled environment on the basis of embedding a small amount of attention modules; can be widely applied in the technical field of computer vision.

[0016] US20210192192 describes a method and an apparatus for facial expression recognition is provided. A terminal device may obtain a face image by detecting an inputted image, determine expression classifications in the face image based on an expression classification standard, obtain expression coefficients of the expression classifications, and recognize expressions in the face image based on the expression coefficients.

[0017] US20230011635, CN112381061 and US20210192192 do therefore perform facial expression recognition, but are vision-based approaches that do not operate in a privacy- preserving manner.

[0018] CN114767112 provides an emotion recognition method, an emotion recognition device and electronic equipment, and relates to the technical field of artificial intelligence, wherein the emotion recognition method comprises the following steps: the method comprises the steps of extracting an RR interval sequence from a heart rate signal of a target object detected based on a millimeter wave radar, and acquiring heart rate features based on the RR interval sequence; extracting a respiratory feature from a respiratory signal of the target object detected based on the millimeter wave radar; acquiring a reflection signal of a face area of the target object based on the reflection signal of the millimeter wave radar to the target object, and extracting expression features according to the reflection signal of the face area; and performing emotion recognition according to the heart rate characteristics, the breathing characteristics and the expression characteristics to obtain an emotion recognition result. According to the embodiment of the invention, the emotion recognition is carried out by combining the heart rate characteristic, the respiration characteristic and the expression characteristic, so that the accuracy of emotion recognition can be improved.

[0019] This therefore uses features from heart rate, breathing, and facial point cloud to output the emotion states and does not perform facial expression recognition. Summary of the Present Invention

[0020] In one broad form, an aspect of the present invention seeks to provide a facial expression sensing method, the method including, in one or more processing devices: acquiring radar data from a radar sensor configured to perform range sensing in an environment including a person; analysing the radar data to determine facial radar data indicative of radar data captured from a face of the person; determining a facial expression by using facial radar data and at least one computational model, the at least one computational model being at least partially indicative of relationships between facial radar data and different facial expressions; and, generating an indication of the facial expression.

[0021] In one embodiment the method includes, in the one or more processing devices: analysing the radar data to perform localisation of the person within the environment; and, determining the facial radar data based on the localisation.

[0022] In one embodiment the method includes, in the one or more processing devices, performing localisation of the person at least in part by analysing the radar data to identify changes indicative of a biometric.

[0023] In one embodiment the biometric includes at least one of: a heat beat; and, respiration.

[0024] In one embodiment the method includes, in the one or more processing devices: analysing changes in the radar data to identify moving objects; filtering the radar data to remove moving objects and thereby identify candidate static objects; and, localising the person using the candidate static objects.

[0025] In one embodiment the method includes, in the one or more processing devices, identifying moving objects at least one of: using an adaptive velocity threshold; and, using changes in range profile.

[0026] In one embodiment the method includes, in the one or more processing devices: determining a range bin for each candidate object; and, localising the person at least in part based on the range bins of the candidate objects.

[0027] In one embodiment the method includes, in the one or more processing devices, filtering the range bin to remove ambient noise.

[0028] In one embodiment the method includes, in the one or more processing devices, analysing the radar data to remove ambient noise and determine range bins using an adaptive cell-averaging constant false alarm rate (CFAR) algorithm.

[0029] In one embodiment the method includes, in the one or more processing devices, analysing the filtered range bins to identify changes indicative of a biometric.

[0030] In one embodiment the radar sensor includes a 2D array of sensing elements that captures radar data over azimuthal and elevational directions and wherein the method includes, in the one or more processing devices: localising the range and azimuthal location of the person; analysing the radar data for the azimuthal location to determine an elevational angle range of a face of the person; and, determining the facial radar data using the elevational angle range.

[0031] In one embodiment the method includes, in the one or more processing devices, determining the elevational angle by: performing high-resolution frequency-wavenumber spectrum analysis to generate enhanced angle range heatmaps; and, using a Gaussian Mixture Model (GMM) to determine the elevational angle range of the face.

[0032] In one embodiment the method includes, in the one or more processing devices: determining one or more features from the facial radar data; and, applying the one or more features to the computational model.

[0033] In one embodiment the method includes, in the one or more processing devices, determining the one or more features using changes in the facial radar data.

[0034] In one embodiment the one or more features include at least one of: heatmaps indicative of changes in facial radar data, the changes in facial radar data being indicative of facial muscle movements; and, Doppler heatmaps.

[0035] In one embodiment the at least one computational model is obtained at least in part by applying machine learning to reference facial radar data acquired from persons having different known facial expressions.

[0036] In one embodiment the at least one computational model is obtained by applying machine learning to heatmaps indicative of changes in the reference facial radar data.

[0037] In one embodiment the at least one computational model is obtained at least in part by applying machine learning to reference image data acquired from persons having different known facial expressions.

[0038] In one embodiment the at least one computational model is obtained by cross domain training using a combination of reference image data and reference facial radar data.

[0039] In one embodiment the at least one computational model is obtained by: applying reference facial radar data to at least one radar data computation model to determine latent radar features; applying reference image data to at least one image data computation model to determine latent image features; using an autoencoder to align the latent radar features with the latent image features; and, using the aligned range features and a pre-trained image classifier model to train the at least one computational model.

[0040] In one embodiment the at least one computational model is obtained using a hybrid loss function that fuses a plurality of loss functions to avoid overfitting.

[0041] In one embodiment the at least one computational model is obtained by: applying azimuthal reference facial radar data to a first model to extract one or more first features; applying elevational reference facial radar data to a second model to extract one or more second features; and, combining the first and second features to generate the latent radar features.

[0042] In one embodiment the method includes, in the one or more processing devices: identifying content being viewed by the person; and, associating an indication of the content and the facial expression.

[0043] In one embodiment the method includes, in the one or more processing devices: transmitting an indication of the content and the facial expression; and, deleting at least one of: radar data; and, facial radar data.

[0044] In one embodiment the radar sensor is at least one of: a radar; a mmwave radar; and, a lidar.

[0045] In one broad form, an aspect of the present invention seeks to provide a facial expression sensing apparatus including: a radar sensor configured to perform range sensing in an environment including a person; and, one or more processing devices configured to: acquiring radar data from the radar sensor; analysing the radar data to determine facial radar data indicative of radar data captured from a face of the person; determining a facial expression by using facial radar data and at least one computational model, the at least one computational model being at least partially indicative of relationships between facial radar data and different facial expressions; and, generating an indication of the facial expression.

[0046] In one broad form, an aspect of the present invention seeks to provide a method of training at least one computational model for use in facial expression sensing, the method including, in one or more processing devices: acquiring reference facial radar data from persons having different known facial expressions; acquiring reference image data from persons having different known facial expressions; and, obtaining the at least one computational model by cross domain training using a combination of reference image data and reference facial radar data.

[0047] In one embodiment the method includes: acquiring radar data from a radar sensor configured to perform range sensing in an environment including a person; and, analysing the radar data to determine the facial radar data, the facial radar data being indicative of radar data captured from a face of the person.

[0048] In one embodiment the method includes, in the one or more processing devices: applying reference facial radar data to at least one radar data computation model to determine latent radar features; applying reference image data to at least one image data computation model to determine latent image features; using an autoencoder to align the latent radar features with the latent image features; and, using the aligned range features and a pre-trained image classifier model to train the at least one computational model.

[0049] In one embodiment the method includes in the one or more processing devices, using a hybrid loss function that fuses a plurality of loss functions to avoid overfitting.

[0050] In one embodiment the method includes, in the one or more processing devices: applying azimuthal reference facial radar data to a first model to extract one or more first features; applying elevational reference facial radar data to a second model to extract one or more second features; and, combining the first and second features to generate the latent radar features.

[0051] In one broad form, an aspect of the present invention seeks to provide a system for training at least one computational model for use in facial expression sensing, the system including one or more processing devices configured to: acquire reference facial radar data from persons having different known facial expressions; acquire reference image data from persons having different known facial expressions; and, obtain the at least one computational model by cross domain training using a combination of reference image data and reference facial radar data.

[0052] It will be appreciated that the broad forms of the invention and their respective features can be used in conjunction and / or independently, and reference to separate broad forms is not intended to be limiting. Furthermore, it will be appreciated that features of the method can be performed using the system or apparatus and that features of the system or apparatus can be implemented using the method. Brief Description of the Drawings

[0053] Various examples and embodiments of the present invention will now be described with reference to the accompanying drawings, in which: -

[0054] Figure 1 is a schematic diagram of an example of a system for performing facial expression recognition;

[0055] Figure 2 is a flow chart of an example of a method for performing facial expression recognition;

[0056] Figure 3 is a schematic diagram of an example of a distributed computer architecture;

[0057] Figure 4 is as schematic diagram of an example of a processing system;

[0058] Figure 5 is a schematic diagram of an example of a client device;

[0059] Figure 6 is a flow chart of a further example of a method for performing facial expression recognition;

[0060] Figure 7 is a flow chart of an example of a method of training a computational model;

[0061] Figures 8A and 8B are a flow chart of a specific example of a method for performing facial expression recognition;

[0062] Figure 9 is a schematic diagram of an example scenario for multimedia IoT applications;

[0063] Figure 10A is a graph showing an example of preliminary results for performing facial expression recognition using point clouds;

[0064] Figure 10B is a graph showing an example of noisy raw mmWave signals;

[0065] Figure 11 is a schematic diagram of an example of a system architecture for performing facial expression recognition;

[0066] Figure 12A is a schematic diagram illustrating an example of the process of face localization;

[0067] Figure 12B is a graph illustrating an example of the ambient noise present in raw mmWave signals;

[0068] Figure 13A is a graph illustrating an example of invalid subject range detection in a Doppler heatmap;

[0069] Figure 13B is a graph illustrating an example of candidate object selection using an adaptive cell-averaging constant false alarm rate (CFAR) algorithm;

[0070] Figure 14A is a graph illustrating an example of heartbeat detection from mmWave signals;

[0071] Figure 14B is a graph illustrating an example of respiration detection from mmWave signals;

[0072] Figure 14C is a graph illustrating an example of heartbeat detection with body motions from mmWave signals;

[0073] Figure 14D is a graph illustrating an example of respiration detection with body motions from mmWave signals;

[0074] Figure 15A is a graph illustrating an example of face localization avoiding body motion;

[0075] Figure 15B is a graph illustrating an example of face matching similarity in elevation;

[0076] Figure 16 is a sequence of images representing comparison between the facial expressions and Doppler heatmaps;

[0077] Figure 17A is a graph illustrating an example of training accuracy comparison on a small-scale mmWave dataset;

[0078] Figure 17B is a graph illustrating an example of loss descent on a small-scale mmWave dataset;

[0079] Figure 18A is a graph illustrating an example of a t-SNE result using KD;

[0080] Figure 18B is a graph illustrating an example of training overfitting;

[0081] Figure 19 is a schematic diagram of an example of a cross domain training process;

[0082] Figure 20A is an image of an example of a facial expression recognition system implementation;

[0083] Figures 20B to 20F are images of example use scenarios for the system of Figure 20A;

[0084] Figure 21A is a graph illustrating an example of face localization performance showing error drift by distances;

[0085] Figure 21B is a graph illustrating an example of face localization performance showing multi-target face localization;

[0086] Figure 22A is a graph illustrating an example of training comparison for different facial expression recognition techniques;

[0087] Figure 22B is a graph illustrating an example of transferability performance for different facial expression recognition techniques;

[0088] Figure 23A is a graph illustrating an example of training ROC curves for different facial expression recognition techniques;

[0089] Figure 23B is a graph illustrating an example of facial expression recognition accuracy for different subjects;

[0090] Figure 24A is an example of a facial expression recognition confusion matrix;

[0091] Figure 24B is a graph illustrating an example of facial expression recognition performance for a user-independent model;

[0092] Figures 25A to 25F are graphs illustrating examples of notable differences amongst subjects for particular facial expressions;

[0093] Figure 26A is a graph illustrating an example of facial expression recognition accuracy for different face orientations and distances;

[0094] Figure 26B is a graph illustrating an example of facial expression recognition accuracy for different distances;

[0095] Figure 27A is a graph illustrating an example of facial expression recognition accuracy for different body motions;

[0096] Figure 27B is a graph illustrating an example of facial expression recognition accuracy for different postures;

[0097] Figure 28 is a sequence of graphs illustrating the impact of regular wearable accessories on facial expression recognition;

[0098] Figure 29 is a graph illustrating an example of the impact of different subjects on facial expression recognition; and,

[0099] Figure 30 is a graph illustrating an example of the impact of illumination on facial expression recognition. Detailed Description of the Preferred Embodiments

[0100] An example of a facial expression recognition system will now be described with reference to Figure 1.

[0101] In this example, the system includes one or more processing devices 100, typically forming part of a monitoring device 120, and a radar sensor 125, such as a mmWave radar.

[0102] In this example, the radar sensor 125 is arranged to perform sensing within an environment E, containing at least one person 101. The nature of the environment will vary depending on the preferred implementation and the usage scenario, but in one example the environment is a room containing audio visual equipment, such as a television, tablet, or other media device, with the person positioned so as to observe the equipment, and hence content presented thereon.

[0103] In this example, the monitoring device 120 includes at least one processing device, such as a microprocessor 121, a memory 122, an external interface 123, and a sensor interface 124, for connecting the monitoring device 120 to the radar sensor 125. In this example the external interface 123 can be utilised for connecting the monitoring device 120 to peripheral devices, such as the communications networks, databases, storage devices, computer systems, media playback devices, or the like. Although a single external interface 123 is shown, this is for the purpose of example only, and in practice multiple interfaces using various methods (e.g. Ethernet, serial, USB, wireless or the like) may be provided.

[0104] In use, the processing device 121 executes applications software stored in the memory 122, allowing the processing device 121 to perform necessary functions, including receivingdata from the radar sensor 125, processing and interpreting the data, and making any required decisions. To achieve these functions, the monitoring device 120 can use multiple processing devices 121, with processing being performed by one or more of the devices 121 as needed. For the purpose of ease of illustration, the following examples will refer to a single processing device, but it will be appreciated that reference to a singular processing device should be understood to encompass multiple processing devices and vice versa, with processing being distributed between the devices as appropriate.

[0105] An example of a method for performing facial expression recognition will now be described with reference to Figure 2.

[0106] In this example, at step 200, the processing device acquires radar data from the radar sensor. The radar data will typically indicate the distance between the radar sensor and objects within the environment, and will extend in azimuthal and elevational directions relative to the radar sensor. In the example of mmWave radar, this is typically achieved using an array of antennas extending laterally and vertically, which detect mmWave signals reflected from the objects within the environment.

[0107] At step 210, the processing device analyses the radar data to determine facial radar data indicative of radar data captured from a face of the person. This can be achieved in a variety of manners, but typically involves analysing the data to identify candidate objects, and then analysing the signals obtained for each candidate object to identify any persons within the environment, which in one example is achieved by identifying reflections indicative of biometrics within the radar data. Facial radar data can then be determined based on geometrical considerations. However, other suitable techniques could be used depending on the implementation.

[0108] At step 220, the processing device determine a facial expression by using facial radar data and at least one computational model. The at least one computational model is at least partially indicative of relationships between facial radar data and different facial expressions, and is typically obtained using a machine learning process, as will be described in more detail below. In one example, this process involves extracting one or more features from the facial radar data, such as Doppler heatmaps, with these features being applied to the model.

[0109] Once a facial expression has been determined an indication of this can be generated at step 230, allowing this to be stored for later use and / or transmitted to external computer systems or similar as required.

[0110] Accordingly, the above described process operates to perform facial expression recognition based on facial radar data captured by a suitable radar sensor. In one particular example, this is performed by using changes in the facial expression data to identify facial movements, which are in turn correlated with different facial expressions using a machine learning trained computational model. This arrangement can achieve a high level of accuracy in facial expression recognition, and avoids the need to use cameras or other imaging devices to capture images of the face of the person, which in turn has associated privacy implications. Furthermore, this approach can use relatively low resolution sensing, such as low resolution mmWave sensing, which in turn allows this to be implemented using existing cheap sensor hardware. These factors result in a system that can accurately sense facial expressions and which can be widely deployed.

[0111] In one example, as will be described in more detail below, the arrangement is used in conjunction with content presentation devices, such as media players, televisions, or the like, allowing a user's reaction to content to be ascertained. This can be used in providing feedback to content creators, for example, by providing advertisers with an indication of the of how users respond to advertising. In this situation, the cheap and easily deployable nature of the hardware make this suitable for integration into media playback devices, such as televisions, set-top boxes, tablets, or the like.

[0112] A number of further features will now be described.

[0113] In one example, the processing device is configured to analyse the radar data to perform localisation of the person within the environment and then determine the facial radar data based on the localisation. Generally, environments will contain multiple objects and surfaces, and it is therefore necessary for the processing device to be able to discriminate between other objects and persons. In one example, this is achieved at least in part by analysing the radar data to identify changes indicative of a biometric, such as a heat beat, respiration, or the like. Being able to identify such biometric signals can enable other objects to be discounted, and allows persons to be located with a high degree of accuracy. Additionally, such signals are typicallyassociated with a person's torso, which can be used to further assist in identifying the location of the person's face.

[0114] In one example, the processing device is configured to analyse changes in the radar data to identify moving objects. In this regard, in general when consuming content, a person will typically be reasonably static relative to the presentation device and so identifying and then removing moving objects can facilitate the process for identifying persons. In this regard, once identified, the processing device filters the radar data to remove moving objects and thereby identify candidate static objects, with these being used to localise the person.

[0115] Identification of moving objects can be achieved using a variety of techniques, such as comparing the range of movement of an object to a threshold, or similar. In one example, this is achieved using an adaptive velocity threshold and / or changes in a range profile, and example of this will be described in more detail below.

[0116] In one example, once moving objects have been excluded, the processing device is configured to determine a range bin for each candidate static object and then localise the person at least in part based on the range bins of the candidate objects. As part of this process, the processing device can be configured to filter the range bin to remove ambient noise, for example using an adaptive cell-averaging constant false alarm rate (CFAR) algorithm. This process can assist in more accurately identifying persons, and specifically allows the filtered range bins to be analysed in order to identify changes indicative of a biometric, specifically to monitor for movement having a frequency corresponding to that of a heart rate or respiration rate.

[0117] As previously mentioned, the radar sensor can include a 2D array of sensing elements, such as antennas, that capture radar data over azimuthal and elevational directions. In this instance, the processing device is configured to localise the range and azimuthal location of the person, based on the location of an identified biometric. Once this has been done, the processing device is configured to analyse the radar data for the azimuthal location to determine an elevational angle range of a face of the person. In this regard, for the majority of the time, when viewing content, the user's face will be located approximately directly above the torso, and so once the biometric has been localised, the processing device can scan radar data capturedabove the biometric to localise the face, and specifically identify an angular range over which the face extends. Facial radar data can then be identified using the elevational angle range.

[0118] The elevational angle range can be determined using any suitable technique and examples include performing high-resolution frequency-wavenumber spectrum analysis to generate enhanced angle range heatmaps and / or using a Gaussian Mixture Model (GMM) to determine the elevational angle range of the face.

[0119] Once the facial radar data has been identified, the processing device is configured to determine one or more features from the facial radar data and then apply the one or more features to the computational model. The nature of the features will vary depending on the preferred implementation, but typically these are based on changes in the facial radar data, and more typically are heatmaps indicative of changes in facial radar data, which are in turn indicative of facial muscle movements. In one specific example, these are in the form of Doppler heatmaps derived from the facial radar data using suitable processing techniques, such as applying an FFT (Fast Fourier Transform) to facial radar data captured over time.

[0120] The computational model is obtained at least in part by applying machine learning to reference facial radar data acquired from persons having different known facial expressions, for example by applying machine learning to heatmaps indicative of changes in the reference facial radar data.

[0121] The nature of the machine learning performed can be of any appropriate form and could include any one or more of decision tree learning, random forest, logistic regression, polynomial regression, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, genetic algorithms, rule-based machine learning, learning classifier systems, or the like. As such schemes are known, these will not be described in any further detail. In one preferred example, the training is performed at least in part using a convolutional neural network (CNN), or other similar approach.

[0122] Typically, facial data collected from persons having known expressions is sparse and expensive to collect, requiring facial radar data to be captured for multiple subjects with multiple different facial expressions. This issue alone can make the system difficult to deploy.

[0123] Accordingly, in one example, the computational model can be obtained at least in part by applying machine learning to reference image data acquired from persons having different known facial expressions. The image data alone is typically not suited for training a model used in analysing facial radar data, so preferably the computational model is obtained by cross domain training using a combination of reference image data and reference facial radar data.

[0124] The cross domain training can be achieved using a variety of techniques. In one example, this is achieved by applying reference facial radar data to at least one radar data computation model to determine latent radar features and also applying reference image data to at least one image data computation model to determine latent image features. An autoencoder can then be used to align the latent radar features with the latent image features, with the aligned range features and a pre-trained image classifier model being used to train the computational model. To ensure this approach is effective, particularly given the sparse nature of the reference facial radar data, the computational model is obtained using a hybrid loss function that fuses a plurality of loss functions to avoid overfitting.

[0125] In one particular example, the computational model is obtained by applying azimuthal reference facial radar data to a first model to extract one or more first features, applying elevational reference facial radar data to a second model to extract one or more second features and combining the first and second features to generate the latent radar features. This can be used to more effectively identify features, particularly taking into account the fact that there may be differences in the resolution of the facial radar data in the azimuthal and elevational directions.

[0126] In one example, the system is configured to identify content being viewed by the person and associate an indication of the content and the facial expression. In this example, the processing device can be configured to transmit an indication of the content and the facial expression, for example, returning this information to an advertiser or other content producer or provider, allowing them to assess feedback based on the person's reactions.

[0127] The processing device can be further configured to delete captured data, such as radar data and more particularly facial radar data, which can help alleviate privacy concerns associated with the capture of such data.

[0128] The radar sensor can be of any appropriate form, and typically includes a radar, such as a mmWave radar, although it will be noted that similar approaches could also be used with other sensing arrangements, such as lidar.

[0129] A specific example of a facial expression recognition system will now be described in more detail with reference to Figures 3 to 6.

[0130] In this example, the system includes a plurality of monitoring devices 120, each of which is used to monitor one or more persons in respective environments. A number of client devices 330 and one or more processing systems 310, such as one or more servers, are provided in communication with the monitoring devices 120 via one or more communications networks 340.

[0131] It will be appreciated that the configuration of the networks 340 are for the purpose of example only, and in practice the monitoring devices 120, client devices 330 and the processing system 310 can communicate via any appropriate mechanism, such as via wired or wireless connections, including, but not limited to mobile networks, private networks, such as an 802.11 networks, the Internet, LANs, WANs, or the like, as well as via direct or point-to-point connections, such as Bluetooth, or the like.

[0132] Whilst the processing system 310 is shown as a single entity, it will be appreciated that in practice the processing system 310 can be distributed over a number of geographically separate locations, for example as part of a cloud-based environment. However, the above described arrangement is not essential and other suitable configurations could be used.

[0133] An example of a suitable processing system 310 is shown in Figure 4.

[0134] In this example, the processing system 310 includes at least one microprocessor 411, a memory 412, an optional input / output device 413, such as a keyboard and / or display, and an external interface 414, interconnected via a bus 415 as shown. In this example the external interface 414 can be utilised for connecting the processing system 310 to peripheral devices,such as the communications networks 340, databases 311, other storage devices, or the like. Although a single external interface 414 is shown, this is for the purpose of example only, and in practice multiple interfaces using various methods (e.g. Ethernet, serial, USB, wireless or the like) may be provided.

[0135] In use, the microprocessor 411 executes instructions in the form of applications software stored in the memory 412 to allow the required processes to be performed. The applications software may include one or more software modules, and may be executed in a suitable execution environment, such as an operating system environment, or the like.

[0136] Accordingly, it will be appreciated that the processing system 310 may be formed from any suitable processing system, such as a suitably programmed client device, PC, web server, network server, or the like. In one particular example, the processing system 310 is a standard processing system such as an Intel Architecture based processing system, which executes software applications stored on non-volatile (e.g., hard disk) storage, although this is not essential. However, it will also be understood that the processing system could be any electronic processing device such as a microprocessor, microchip processor, logic gate configuration, firmware optionally associated with implementing logic such as an FPGA (Field Programmable Gate Array), or any other electronic device, system or arrangement.

[0137] As shown in Figure 5, in one example, the client device 330 includes at least one microprocessor 531, a memory 532, an input / output device 533, such as a keyboard and / or display, and an external interface 534, interconnected via a bus 535 as shown. In this example the external interface 534 can be utilised for connecting the client device 330 to peripheral devices, such as the communications networks 340, databases, other storage devices, or the like. Although a single external interface 534 is shown, this is for the purpose of example only, and in practice multiple interfaces using various methods (e.g. Ethernet, serial, USB, wireless or the like) may be provided.

[0138] In use, the microprocessor 531 executes instructions in the form of applications software stored in the memory 532 to allow required processes to be performed. The applications software may include one or more software modules, and may be executed in a suitable execution environment, such as an operating system environment, or the like.

[0139] Accordingly, it will be appreciated that the client devices 330 may be formed from any suitable processing system, such as a suitably programmed PC, Internet terminal, lap-top, or hand-held PC, and in one preferred example is either a tablet, or smart phone, or the like. Thus, in one example, the client device 330 is a standard processing system such as an Intel Architecture based processing system, which executes software applications stored on non- volatile (e.g., hard disk) storage, although this is not essential. However, it will also be understood that the client devices 330 can be any electronic processing device such as a microprocessor, microchip processor, logic gate configuration, firmware optionally associated with implementing logic such as an FPGA (Field Programmable Gate Array), or any other electronic device, system or arrangement.

[0140] For the purpose of the following examples, it is assumed that one or more processing systems 310 are servers, which communicate with the monitoring devices 120 and client devices 330 via a communications network, or the like, depending on the particular network infrastructure available. The servers 310 typically execute applications software for performing required tasks including receiving indications of facial expressions provided by the monitoring devices 120, and provided access to this information via the client devices 330. It will be appreciated that actions performed by the servers 310 are performed by the processor 411 in accordance with instructions stored as applications software in the memory 412 and / or input commands received from a user via the I / O device 413, or commands received from the client device 330.

[0141] It will also be assumed that users interact with the client device 330 via a GUI (Graphical User Interface), or the like presented on a display of the client device 330, and in one particular example via a browser application that displays webpages, or an App that displays relevant information. Actions performed by the client devices 330 are performed by the processor 531 in accordance with instructions stored as applications software in the memory 532 and / or input commands received from a user via the I / O device 533.

[0142] However, it will be appreciated that the above described configuration assumed for the purpose of the following examples is not essential, and numerous other configurations may be used. It will also be appreciated that the partitioning of functionality between the client devices 330, and the servers 310 may vary, depending on the particular implementation.

[0143] An example of the process for recognising facial expressions will now be described in more detail with reference to Figure 6.

[0144] In this example, at step 600, the monitoring device 120 acquires radar data from the radar sensor 125, with the processing device 121 analysing this to localise any persons within the environment at step 610, typically by detecting patterns of movements corresponding to biometrics, such as a heartbeat, or respiration.

[0145] At step 620, the processing device 121 identifies facial radar data, typically by performing a search in an elevational direction, based on the localisation of any persons in the azimuthal direction.

[0146] At step 630 the processing device 121 generates a Doppler heatmap based on changes of the facial radar data captured over multiple measurements performed by the radar sensor 125. For example, this process might involve examining multiple measurements within a sliding window, based either on a time period or number of measurements, and using changes in the facial radar data to generate a Doppler heatmap indicative of facial movements using an FFT or similar, with this being updated as further measurements are performed.

[0147] At step 640, the processing device 121 applies to the Doppler heatmap to the computational model, which is typically stored locally on the monitoring device 120, using this to generate an indication of a facial expression at step 650.

[0148] At the same time this is performed, at step 660, the processing device 121 can optionally identify any content being displayed to the person, for example by communicating with a display device, such as a set-top box or television. This information, typically in the form of a content identifier and optional time stamp to indicate a specific part of the content, can then optionally be uploaded to the server 310 at step 670, together with an indication of the corresponding facial expression allowing this information to be used by content providers, for example by accessing the information using a client device 330. This allows users of the client device to receive information regarding the reaction of persons viewing the content, for example, allowing them to assess whether the content has been well received.

[0149] An example of a process for training a computational model will now be described with reference to Figure 7.

[0150] In this example, the process typically includes acquiring reference facial radar data at step 700 with an indication of a corresponding facial expression, being determined at step 710. This can be achieved in any suitable manner, but will typically involve using a monitoring device 120 to collect the facial data, with this being provided to a processing system, such as a server 310 for model training. Information regarding the facial expression can be entered manually, for example using a client device 330, or the like, with this being performed contemporaneously with collection of the radar data (or alternatively by imaging the face of the person as the facial radar data is captured and entering the information later based on the image data).

[0151] At step 720, the processing system extracts latent radar features from the facial radar data. This for example can involve determining Doppler heatmaps, using the techniques described above, and then using feature extraction techniques, such as applying the heatmaps to a CNN.

[0152] Meanwhile, at step 730 equivalent image data is collected, for example using an imaging device or similar, with associate facial expressions being determined at step 740. It will be appreciated that in this case, the facial expression can be determined manually by viewing the image data. Again latent image features are determined at step 750, for example by applying the image data to a CNN.

[0153] At step 760, feature alignment is performed using an autoencoder, to align the facial radar data features with the image data features. This allows the image features to be used to train a computational model at step 770, as will be described in more detail below.

[0154] A further more detailed example of the process for performing facial expression recognition will now be described with reference to Figures 8A and 8B.

[0155] In this example, at step 800, the monitoring device 120 acquires radar data from the radar sensor 125, and analyses changes in the radar data at step 805, with this process being repeated continuously as further radar data is captured. The radar data is analysed in orderidentify and remove moving objects at step 810, which in one example is achieved by identifying objects whose range changes by more than a determined amount. This can be achieved using an adaptive velocity threshold and / or changes in range profile, as will be described in more detail below.

[0156] For any static objects, range bins are established at step 815, with these being filtered at step 820, to remove ambient noise, which could be achieved using adaptive cell-averaging constant false alarm rate (CFAR) algorithms, or the like.

[0157] At step 825, the filtered range bins are analysed in order to identify any range bins including movement corresponding to a biometric, such as a heartbeat or respiration, with results of this being used to localise persons within the environment at step 830.

[0158] At step 835, faces of the persons are identified, with this typically involving using the azimuthal location of the identified persons, and then scanning upwardly in an elevational direction, to identify the face, with this process optionally employing the use of a face-matching mechanism based on a Gaussian Mixture Model (GMM).

[0159] Facial Doppler heatmaps are generated at step 840 based on changes in the facial radar data over time, with these being applied to the computational model at step 845, thereby allowing facial expressions to be determined at step 850.

[0160] This information can be stored and transmitted at step 855, with any captured data being deleted at step 860.

[0161] A specific example of the facial recognition process will now be described. INTRODUCTION

[0162] Millimeter wave (mmWave) sensing has recently been popular due to its high bandwidth and robustness. Compared with Wi-Fi and ultrasound sensing, mmWave sensing provides higher signal resolution, hence being capable of detecting subtle movements. In addition, it can detect multiple targets due to its high range resolution. mmWave sensing is also illumination free (i.e., working in the dark) with fair penetration (e.g., see-through glasses and masks)

[0054] , hence being promising for a wide range of applications. Figure 9 shows anexample of a scenario for multimedia IoT applications, in which mmWave radar is used to capture facial expressions as will now be described.

[0163] Facial expression usually triggers a series of facial muscle movements across multiple facial areas (e.g., eyes, forehead, nose, cheek, lip and mouth), representing as critical spatial information in mmWave signals. To acquire spatial information, mmWave radar integrates an antenna array with multiple transmitters (TX) and multiple receivers (RX) to enable MIMO (i.e., multi-input and multi-output) in improving angular resolution. A de-facto way is to acquire a set of point clouds generated by mmWave radar. However, due to the limited number of antennas available on an off the-shelf mmWave radar (e.g., TI IWR1843BOOST mmWave radar has 3 TX and 4 RX resulting up to 12 virtual antennas), the angular resolution is limited to 15-degree in azimuth only, resulting in limited number of points in resulting point clouds. Also, to improve signal-to-noise ratio, point clouds generated from raw mmWave signals are intently merged by built-in algorithms, hence yielding notable sparsity.

[0164] Preliminary studies described below demonstrate it is not feasible to detect facial muscle movements using sparse point clouds. Point clouds may be enhanced using advanced mmWave radar with high angular resolution. However, advanced radar is usually bulky and more expensive (e.g., 10x) compared to commercial grade radar and is typically used in high- end automotive. Supervised learning from cross-domain (e.g., colabeled LiDAR dataset) can also be used to generate dense point clouds, but its performance relies heavily on largescale training datasets, which may be impractical for FER due to lack of mmWave datasets.

[0165] As an alternative to using sparse point clouds, raw mmWave signals received from multiple antennas contain rich Doppler information, which can be used to generate spatial information. Raw mmWave signals contain body motion information and ambient noise due to background moving objects, diffraction, and complex specular reflections. Also, facial muscle movements are relatively subtle, e.g., 5mm for cheek and 6mm for month by a "happiness" expression of adult, making it difficult to be distinguished from ambient noise. Beamforming uses narrow beams which focus on a relatively small area, hence avoiding ambient noise to some extent. However, this limits the ability to detect multiple targets at a time, and ends up with much reduced spatial information since all antennas are phased to narrow the beam compromising on angular resolution.

[0166] In contrast, in one example, the approach described below converts the problem to a spatial localization problem. This locates each subject by verifying their biometric information (i.e., heart rate and respiration) to eliminate ambient noise of static / dynamic objects. Spatial facial information is then extracted by filtering out irrelevant body motions. With spatial facial information obtained, the correlation between facial muscle movements and features extracted from spatial facial information can be determined. Deep Learning (DL) can potentially achieve high accuracy and better robustness due to its automated feature extraction capability. However, deep learning usually requires a large amount of training data to train an effective model, and small-scale training dataset may easily end up with poor performance. In reality, collecting large-scale mmWave dataset can be very costly due to labelling efforts and privacy concerns. Accordingly, in one example, a rich image dataset available for FER is leveraged by applying cross-domain transfer learning that enables model knowledge transformation from the image domain to the mmWave domain.

[0167] Accordingly, in one example, a mmWave radar based FER system (hereafter mmFER) is provided that extracts and analyzes subtle facial muscle movements to recognize facial expressions for multiple users. mmFER can recognize multiple standard facial expressions, delivering valuable assessment of users’ engagement and reactions to multimedia IoT applications.

[0168] To detect subtle facial muscle movements in raw mmWave signals, the system can use a dual-locating approach based on MIMO to locate both subjects of interest and the facial areas of each subject. Specifically, in one approach, the system first locates subjects of interest (e.g., who stay relatively stationary to watch TV) marked as anchor points in the azimuth detection range (i.e., horizontal plane). This is achieved by sensing biometric information (i.e., heart rate and respiration) of subject, filtering out background noise (e.g., complex specular reflections by appliances and walls) and dynamic objects (e.g., people moving around). A face-matching mechanism based on a Gaussian Mixture Model (GMM) can then be used to locate each subject’s facial angle range in the elevation detection range (i.e., vertical plane), avoiding body motions.

[0169] To fully explore the transferability from image to mmWave, cross-domain transfer pipeline, namely crosstransfer, is provided using a pre-trained FER image model to train anmmWave model. Especially, a hybrid learning loss function is designed that comprehensively fuses a set of loss functions to address the training over-fitting issue due to small-scale mmWave datasets. Autoencoder based feature alignment mechanism is used that learns the transition of latent features to progressively reshape complex mmWave data, eliminating the effect of data heterogeneity.

[0170] Using such approaches, mmFER can extract spatial information of facial muscle movements from raw mmWave signals. With cross-transfer, mmFER enables an effective and safe model knowledge transformation for mmWave-based FER.

[0171] A fully implement mmFER using an off-the-shelf mmWave radar (i.e., TI IWR1843BOOST) will now be described, with comprehensive evaluations with 10 subjects for a set of facial expressions in real-world settings. Results show that mmFER achieves an accuracy of 84.48% in a subject to-radar distance between 0.3 and 1.5m and an accuracy of 80.57% when distance increases to 2.5m.

[0172] Thus, the system can provide an mmWave radar based FER system that detects subtle facial muscle movements associated with raw mmWave signals for multimedia IoT applications. This can employ a dual-locating approach to accurately locate on subjects’ faces in space, and extract subtle facial muscle movements from noisy raw signals. A cross-domain transfer pipeline, i.e., cross-transfer, can be used to enable an effective and safe model knowledge transformation for mmWave-based FER with superior model performance.

[0173] An off-the-shelf mmWave radar based mmFER implementation can be employed with extensive experiments showing that that mmFER is resilient to various real-world settings with high accuracy and robustness, outperforming the baselines.

[0174] The proposed mmWave radar approach easily removes privacy concern and illumination constraints, and robustly works in the scenario of wearing various accessories (e.g., mask). It is also superior to Wi-Fi and ultrasonic approaches, providing a higher signal bandwidth with a longer detection range and multi-target capability.

[0175] While the following focuses on enabling mmWave-based FER for multimedia IoT applications, mmFER has opened possibilities to be widely applied in different domains suchas recommendation systems, healthcare, augmented reality / virtual reality (AR / VR) systems, education, and more, providing valuable user feedback and survey.

[0176] For instance, mmFER can be used in a wide range of recommendation systems to help sense users’ preferences and reactions in a privacy-preserving manner. In healthcare systems, mmFER is able to offer medical professionals timely feedback about the mental state of patients (e.g., with depression). In addition, mmFER can provide a robust way to understand users’ attention and intent in an indoor or outdoor environment, improving the user experience of AR / VR systems. PRELIMINARY Principles of MIMO in mmWave Radar.

[0177] mmWave radar has a linear RX antenna array which enables MIMO to estimate target’s angle of arrival (AoA), hence providing spatial information for detecting multiple facial areas. In principle, since RX antenna array is designed with an inter-antenna distance (e.g., 2.5mm on TI IWR1843BOOST), the reflected mmWave signals from a target travel different distances to reach each RX antenna, yielding phase difference which can be used to estimate the AoA of signals. Theoretically, to enrich spatial information for detecting multiple facial areas, the angular resolution governed by the number of RX antennas could be increased. However, commercial mmWave radars only integrate a limited number of antennas, e.g., TI IWR1843BOOST has a 3x4 antenna array which achieves 12 virtual RX antennas with a 2- dimensional 2x8 antenna layout, yielding a limited 58-degree and 15-degree angular resolution in elevation and azimuth, respectively. Set-up.

[0178] To investigate the feasibility of mmWave sensing for FER, preliminary experiments were conducted using a commercial mmWave radar (TI IWR1843BOOST). This involves a movie watching scenario using a 27-inch screen with the radar placed directly below the screen (1.1m height the ground). A subject sits 1m away from the screen and watches movies. The radar configuration is shown in Table 1 below.

[0179] A "surprise" facial expression is used for this study since it triggers a relatively large range of facial muscle movements. Since the radar has a limited angular resolution especially in elevation (i.e., 58-degree), the typical setup may cause loss of useful spatial information in elevation for detecting facial areas. Hence, the radar is placed upright to switch the 15-degree angular resolution from azimuth to elevation. Sparse Point Clouds.

[0180] An experiment was performed to examine the feasibility of using point clouds. In this regard, point clouds represent all motions in space over time. A set of point clouds were generated based on raw mmWave signals using a de-facto approach. Figure 10A presents the changes of motion points in three consecutive time frames (e.g., t − 1 to t + 1) with an interval of 0.5s. The results show that point clouds are sparse and largely contain irrelevant motions or ambient reflections. Block 1081 marks the location of the subject’s face as the ground truth. In particular, the zoom-in figure indicates that mmWave radar can detect the change of point clouds caused by facial muscle movements, but point clouds are highly sparse and only one point is available in each time frame. Thus, it is infeasible to leverage on sparse cloud points to obtain spatial facial information for FER. Challenges in Raw mmWave Signals

[0181] Instead raw mmWave signals can be used to extract spatial facial information. Given a pre-defined subject’s position (i.e., a set of range bins), an angle FFT is applied to transform time-domain raw data within a 1s time window to spatial data (i.e., containing range and angle information). To understand the impact of body motion, a subject performs leg shaking in this experiment. To obtain the ground truth of facial angle range, vision-based face detection is performed to locate face in space. According to the ground truth lines shown in Figure 10B, it is feasible to locate the spatial facial information in angle-range heatmap (e.g., marking with the green block). However, due to the limited 15-degree angular resolution, it is difficult toprecisely locate facial areas and avoid irrelevant spatial information using angle FFT. Moreover, MIMO-enabled spatial data contain massive ambient noise represented as ambiguity blocks, and the spatial information of body motion block can be observed in the heatmap. Thus, it is challenging to extract subtle spatial information of facial expressions from raw mmWave signals. mmFER DESIGN

[0182] Figure 11 presents an overview of mmFER system architecture. Due to ambient noise existing in raw mmWave signals, the approach uses spatial face localization achieved by a dual-locating approach to extracting spatial facial information. This approach takes raw mmWave signals as input and outputs an angle (A) and range (R) heatmaps of multiple facial areas, as shown in Figure 12A. A cross-transfer pipeline is used, feeding the outputs into an mmWave FER model for classification. Dual-locating Approach Challenge of Massive Ambient Noise

[0183] Facial muscle movements caused by facial expressions are subtle in millimetre level. mmWave radar is capable of detecting millimetre level movements, but raw mmWave signals received indoors usually contain massive ambient noise due to dynamic and stationary objects, e.g., people moving around, multi-path reflections by home appliances and walls.

[0184] To further understand the impact of ambient noise, an experiment was conducted in a living room (e.g., 2.9m x 4.2m) with the setup shown in Figure 12A. A fan was placed 1.5 m behind the subject, and another subject walked randomly in the room. Figure 12B shows the range profile heatmap generated from raw mmWave signals, and it clearly shows that mmWave signals contain massive ambient noise.

[0185] To address, based on the MIMO capability on mmWave radar, the dual-locating approach is used, which consists of a two-step process to first locate subjects, then locate the facial areas of each subject, as shown in Figure 12A. Subject Localization

[0186] In Step-1, the dual-locating approach locates subjects of interest (i.e., who stay stationary to watch TV), marked as anchor points in the azimuth detection range, and eliminates background noise and dynamic objects. Dynamic Object Removal

[0187] Since dynamic objects (e.g., people walking) have speed information, range Doppler information can be used to distinguish the difference between stationary and dynamic objects in velocity for range detection. In a simple way, a fixed speed threshold (e.g., 0.11m / s) can be used to filter out dynamic objects which move faster than this threshold.

[0188] However, applying range Doppler information for computing velocity is very sensitive to speed, often resulting in unsafe object removal (i.e., subject removed due to his / her body motions). In reality, a subject may have body motions at different speeds, e.g., 0.4-0.6m / s for leg shaking, and 0.9-1.2m / s for waving hands.

[0189] Due to radar’s Doppler widening effect (i.e., the area of Doppler spectrum is widened by the changes of object’s speed and range), and so the Doppler spectrum of body motions may largely overlap with that of facial movements (noting that body movement may have a higher speed than facial movement). Due to the overlapping, removing body motion spectrum could in loss of facial movement spectrum. Hence, a fixed threshold method may lead to subject localization failure.

[0190] To enable a safe dynamic object removal, range profile information is used directly to estimate moving objects if they trigger range shift. Specifically, a time difference function is defined as: ^^ ^^ ^^ ^^ ( ^^) = ^^ ^^ ^^ − ^^ ^^ ^^− ^^ where ^^ ^^ ^^ is range profile on time t, and n is time slot.

[0191] An adaptive velocity threshold is formulated as ^^ ^^ = Δ ^^ ^^ ^^ / ^^× ^^, where Δ ^^ ^^ ^^ is range resolution (e.g., 4.2cm), ^^ is frame periodicity (e.g., 10ms). In addition, the range of dynamic object is in: Where:

[0192] ^^ ^^ denotes a find peak function based on ^^ ^^ ^^ ^^ ( ^^), and ^^ denotes each dynamic object.

[0193] Once the time difference function detects peaks in ^^ ^^ ^^ ^^, it indicates in presence of moving objects whose velocity is greater than the threshold, and a set of range bins (i.e., object positions) will be computed for accurate dynamic object removal. Conversely, the objects whose velocity is less than the threshold are recognized as stationary to safely protect subjects with body motions.

[0194] Thus, dual-locating can safely remove dynamic objects and preserve subjects of interest based on the range shift over time. Candidate Range Localization

[0195] After removing dynamic objects, subjects of interest (i.e., stationary objects) can be located and marked by a set of range bins, as candidates for FER. Due to complex indoor reflections, raw mmWave signals contain massive ambient noise which impact candidate range localization. Existing works obtain a resilient noise threshold by the cell-averaging constant false alarm rate (CFAR) algorithm described in V Shrivathsa.2018. Cell Averaging-Constant False Alarm Rate Detection in Radar. International Research Journal of Engineering and Technology (IRJET) 7 (2018), 2433–2438, based on range Doppler information.

[0196] When applied in this case, range Doppler information may be easily corrupted by ambient motion noise (e.g., regular vibration caused by fan or air conditioner) due to the Doppler widening effect.

[0197] Figure 13A shows that the range frequency caused by a running fan is largely widened in the range Doppler heatmap, and such enlarged noise frequency notably increases the noise threshold of CFAR. With the increased noise threshold, the selected range bins of subject using CFAR in lines 1382 is reduced by 60% compared to the ground truth in white lines 1383. Reduced spatial information may easily result in invalid removal of facial areas of subjects.

[0198] Alternatively, range profiles can be leveraged generated by the first FFT to detect the presence of objects with higher amplitudes than the noisy background. To accurately detect subject range (i.e., no range bin reduction) and remove ambient noise, an adaptive CFAR witha body knowledge based Gaussian distribution is used. Accurately selecting subject’s range (i.e., covering subject’s body) is the key to obtain spatial facial information.

[0199] The basic idea is to build a Gaussian distribution ^^ ( ^^, ^^) using reported body shape statistics to improve CFAR with a robust noise threshold function. Specifically, a range profile RP is defined as input of C(·) (i.e., CFAR), and the output of CFAR denotes a set of range bins formulated as ^^( ^^ ^^ ^^) = ^^ ^^ ^^ - ^^̂ ^^, where ^^̂ ^^ is an adaptive resilient noise threshold calculated from Eq.1.

[0200] where NCndenotes number of noise cells. NCgdenotes number of guard cells formulated as N ^^ ^^ = 1 / 2 ( ^^+3 ^^ / Δ ^^ ^^ ^^), where ^^ and ^^ obtained from ^^ ( ^^, ^^), and Δ ^^ ^^ ^^ is range resolution (e.g., 4.2cm). Figure 13B demonstrates that subject’s range bins can be successfully located without information loss, and most of the ambient noise can be effectively removed using the adaptive CFAR. Biometric Verification

[0201] After candidate range localization, a number of stationary objects may be selected as candidates which may still contain ambient noise (e.g., fan and wall) as shown in Figure 13B. To further validate subjects of interest in a list of candidates, it is possible to leverage on biometric information (i.e., heartbeat or respiration) to effectively distinguish human from other objects. Given a range bin of each candidate, first access phase change defined as ( ^^ ^^ ^^, ^^) in mmWave signals over time. A 4th-order Butterworth band-pass filter is then applied to extract heartbeat and breath signals from the phase change.

[0202] To compare with the ground truth reported in existing vital sign detection works, chest displacement is calculated ΔR (mm) with phase change Δ ^^ using Δ ^^ = (4π / λ)ΔR, where λ is the mmWave wavelength. Results can be verified with the chest displacement metrics (e.g., 0.1-0.5mm for heartbeat and 1-12mm for respiration).

[0203] As shown in Figures 14A and 14B, results of both fan (0.054mm for heartbeat and 0.109mm for respiration) and wall (0.018mm for heartbeat and 0.073mm for respiration) are invalid compared to the displacement metrics, while the subject is successfully verified by accessing both heartbeat (0.15mm) and respiration (1.22mm). Besides, since body motions may remain at this step, the impact of these motions (e.g., playing games, typing, and leg shaking) on heartbeat and respiration detection can be tested.

[0204] The results from Figures 14C and 14D indicate that the phase change of both heartbeat and respiration can be accurately detected with no impact, e.g., the verified displacement of 0.36mm on average for heartbeat with motions and 1.54mm on average for respiration with motions. After biometric verification, dual-locating is able to accurately locate subjects of interest with correct bins in azimuth. 3.1.2 Face Localization

[0205] In Step-2, given a subject’s location in azimuth (i.e., anchor point), dual-locating locates subject’s face with a specific angle range in the elevation detection range while avoiding irrelevant body motions. As aforementioned, due to the limited angular resolution of 15-degree on the mmWave radar, it is difficult to locate small facial areas by angle range. To achieve high-resolution AoA estimation (e.g., theoretically 1-degree in azimuth), the approach used in J. Capon.1969. High-resolution frequency-wavenumber spectrum analysis. Proc. IEEE 57, 8 (1969), 1408–1418, is implemented to generate enhanced angle range heatmaps.

[0206] With enhanced angular information, a face-matching mechanism that uses a greedy search based on GMM Ismail Shahin, Ali Bou Nassif, and Shibani Hamsa. 2019. Emotion recognition using hybrid Gaussian mixture model and deep neural network. IEEE access 7 (2019), for matching similarity of facial features in elevation. Once facial features are matched with angle range, dual-locating extracts the facial areas in the enhanced angle range heatmap (i.e., spatial information) for the next process. Face-matching Mechanism

[0207] Due to the advanced capability of approximating spectral features using GMM as described in Huining Li, Chenhan Xu, Aditya Singh Rathore, Zhengxiong Li, Hanbin Zhang, Chen Song, Kun Wang, Lu Su, Feng Lin, Kui Ren, and Wenyao Xu. 2020. VocalPrint:Exploring a Resilient and Secure Voice Authentication via MmWave Biometric Interrogation (SenSys’20), a GMM model is trained on pre-processed mmWave dataset ^^ ^^ ^^ ^^ with manual labels for generalization performance. Since the spectral features of facial muscle movements represent as a set of angle range heatmaps in a time slot (e.g., 3s) with rich information in time- domain, the angle range heatmaps can be transformed to angle time heatmaps by collapsing the range information to enhance the features.

[0208] x ^^ is defined as input data of GMM and is obtained from the angle time heatmap on ^^- degree. To measure the similarity between input x ^^ and output by GMM, cosine similarity described in Hieu V Nguyen and Li Bai. 2010. Cosine similarity metric learning for face verification. In Asian conference on computer vision, is used as the loss function. The objective is to locate facial angle range ^^ by maximizing the similarity between the template distribution ^^ ^^ ^^ ( ^^ ^^ ^^ ^^) and the searched distribution ^^ ^^ ^^ ( ^^ ^^), as formulated in Eq.2.

[0209] where ^^ denotes an experimental offset for angle range compensation, and ^^ ^^ ^^ ^^ is the optimized facial angle range. this step,

[0210] In particular, FOV is a 120-degree search grid in space. Figure 15B presents the greedy search process by maximizing similarity, and face-matching can locate subject’s face with maximum similarity along with related angle range (in both azimuth and elevation). Figure 15A shows that face-matching can accurately locate face with angle range, and effectively avoid body motions (i.e., playing games and leg shaking). Figure 16 demonstrates the outputs of dual-locating, comparing to different facial expressions. Cross-transfer Pipeline Challenges of Small-scale Dataset.

[0211] Collecting a largescale mmWave dataset to train a FER model can be very costly in reality due to high labor cost and user privacy, while smallscale training dataset may lead to poor performance. To fully understand this issue, both mmWave and video datasets with 10 subjects (e.g., 40 samples per expression per subject) in a watching-iPad scenario, as shown inFigure 20B, for the apparatus of Figure 20A. To obtain the ground truth, a pre-trained ResNet18 (i.e., image model) was used, and fine-tuned on the collected video dataset. A conventional supervised approach was used with the manual truth labels (i.e., T-label) to train an mmWave model after the dual-locating process.

[0212] Figure 17A reveals that the training accuracy of T-label severely degrades (i.e., 45.71%) compared to the ground truth (i.e., 95.28%). The underlying impediment is that T-label learns insufficient latent features based on a much limited data distribution provided by the small- scale training dataset. The key idea of cross-domain transfer learning is to transform domain- invariant latent features learned from a well-developed source domain (e.g., image model) to a limited target domain (e.g., mmWave model), enhancing latent feature extraction.

[0213] A Knowledge Distillation (KD) based cross-domain approach is applied (i.e., teacher- student learning scheme described in Lin Wang and Kuk-Jin Yoon. 2021. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)) to both mmWave and video datasets, with the image model used as a teacher and the mmWave model as a student. As shown in Figure 17A the KD based approach achieves a higher training accuracy (i.e., 74.38%). However, the KD based approach suffers from suboptimal training performance, yielding a critical accuracy gap between KD and the ground truth. The following two issues remain as impediments. Training Over-fitting

[0214] Figure 17B shows that while KD train loss descent performs well, KD test loss notably fails to decrease. The result demonstrates the training over-fitting issue (i.e., unsafe training) that the model is too closely aligned to the small-scale training dataset, hence resulting poor generalization on testing dataset. To further understand the issue, t-distributed stochastic neighbor embedding (t-SNE) (i.e., clustering analysis for loss optimization) is employed based on the Euclidean distance (ED) function to visualize the distribution of the latent features for six expressions using both image and KD based mmWave models. As shown in Figure 18A (e.g., each color represents an expression), the feature distribution using a KD based mmWave model is messy and scattered compared to the ground truth (i.e., clustered nicely using theimage model), hence it implies that the underlying loss function of KD may perform poorly based on small-scale training dataset. Domain Shift

[0215] Based on transferability metrics (i.e., distance based similarity analysis between domains) described in Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long.2022. Transferability in Deep Learning: A Survey. ArXiv (2022), it is observed that there is a notable domain shift (i.e., lack of similarity) between the feature distributions using both models shown in Figure 18A.

[0216] Due to high data heterogeneity (i.e., different raw data structures and shapes) between image and mmWave, model performance may easily degrade if the latent features between both domains are not well aligned (i.e., domain shift). A keypoint-based approach (Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Throughwall human pose estimation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7356– 7365) is used to extract relevant key points (i.e., less data heterogeneity) from images to achieve image-to-sensing transformation. To validate its effectiveness, Mediapipe (Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. 2019. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172 (2019).) is used to generate facial landmarks from images and use a keypoint-based approach to train an mmWave model, but the results reveal that this approach still suffers from training over-fitting caused by the small-scale dataset, severely degrading model accuracy (i.e., 62.38%) shown in Figure 18B.

[0217] A cross-transfer pipeline is therefore used to enable effective and safe model knowledge transformation from image to mmWave. Specifically, to address training over-fitting, a hybrid learning loss function is designed that comprehensively fuses a set of loss functions including a supervised loss by mmWave labels, a Kullback–Leibler (KL) divergence loss inspired by KD, and a contrastive loss based on positivenegative correlation to achieve superior model performance.

[0218] To eliminate the impact of data heterogeneity, an autoencoder is developed based feature alignment mechanism that learns the transition of latent features to progressively help align both domains, thus improving transferability. Training Scheme Design

[0219] Figure 19 presents the detailed training scheme design of the cross-transfer pipeline.

[0220] The pre-trained ResNet18 is kept as the image model, and fine-tuned on the collected face images for learning the latent features. This is frozen (i.e., model inference only) as a learning anchor point throughout the process of cross-transfer pipeline. Once mmWave data is processed by dual-locating, all of the spatial facial information will be used as the input of the mmWave model. Feature Alignment

[0221] Since mmWave data input consists of both azimuth and elevation heatmaps with different data shapes, simply combining them may result in loss of key features. Hence, two separated blocks are designed with four convolutional neural network (CNN) layers for each to reshape and explore the key features, avoiding information loss. After this, both outputs of the heatmaps can be safely summed to prepare for the next process, as shown in Figure 19. To align the latent features with different data structures for image to mmWave, a CNN-based autoencoder is designed to essentially learn from the transition of both latent feature distributions during training. In theory, autoencoder is advocated to fully explore the latent information to improve training performance. In mmFER, due to data heterogeneity, the transition of feature distributions between both domains can be very complex, therefore fully exploring useful latent features is the key. The autoencoder is designed to not only reshape the dimensionality of latent features (i.e., data shape alignment), but also enhance transferability to contribute in loss optimization for training performance improvement. Since the image model is set as the learning anchor point, the proposed autoencoder is placed in an mmWave model as a middleware to reshape the latent features from the mmWave model, aligning with the feature shape in the image model presented in Figure 19. Hybrid Learning Loss

[0222] As aforementioned, existing cross-domain approaches may suffer from training overfitting due to small-scale mmWave dataset. The underlying impediment is due to an ineffective design of loss function. To address, a hybrid learning loss function is used to improve the loss optimization for image to mmWave. Intuitively, according to the principle of loss optimization, if the feature distribution of each class is more tightly clustered (i.e., smaller intra-class distance) and the distance between the feature distribution of each class pair is farther apart (i.e., larger inter-class distance), it will result in much better training performance (i.e., high accuracy and generalization). Inspired by this principle, the hybrid learning loss function includes three key elements. Cross Entropy Loss

[0223] Based on the T-label approach with manual truth labels, cross entropy loss ^^ ^^ ^^ is calculated with mmWave data ^^ ^^ and its ground truth label ^^ ^^ using Eq.3.

[0224] where ^^ is the cross entropy loss function, ^^ ^^ ( ^^) and ^^( ^^) denote related probability computed by an mmWave model. KL Loss

[0225] Referring to accuracy improvement using KD in Figure 17A, KL divergence is used as one of key metrics to measure the difference between the distribution of logits (i.e., ^^ ^^ and ^^ ^^) computed by both image and mmWave models, hence the KL loss ^^ ^^ ^^ is formulated in Eq.4.Contrastive Loss

[0226] Based on the principle of loss optimization, contrastive loss Lcontrast is designed to essentially optimize two distance-based metrics in cross-domain. The objective is to first minimize the distance between the latent feature distributions that have the same label (i.e., intra-class distance) marked as positive pair, and then maximize the distance between thefeature distributions that have different labels (i.e., inter-class distance) marked as negative pair, in training image to mmWave.

[0227] The positive-negative correlation is visualised as a 2-color matrix in Figure 19, in which the green diagonal line denotes all positive pairs that each xm,Nin the mmWave domain has the same label (e.g., "happiness") with each ^^ ^^,^^ in the image domain, where N denotes the number of classes, and the rest in the matrix denotes all negative pairs. Hence, formulating Lcontrastin Eq.5.

[0228] where ^^υ+and ^^υ−denote the positive and negative inputs in an image model, respectively. Also, fm, ^^υ−,^^and ^^υ+denote the sets of latent features computed by input , ^^υ−,^^and ^^υ+, respectively.

[0229] In particular, a regularizor ^^ ^^ is designed to further contribute optimizing the negative pairs:

[0230] where ^^ denotes a standard ED function, and ^^ denotes a default SoftMax function.

[0231] Combined with all elements, the hybrid learning loss function is formulated as ^^ℎ ^^ ^^ ^^ ^^ ^^ = ^^ ^^ ^^ + ^^ ^^ ^^ + ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^. In short, cross-transfer pipeline enables an effective and safe model knowledge transformation for mmWave-based FER. EVALUATION System Implementation

[0232] Figure 21A gives an overview of an mmFER system setup. This is implemented using an off-the-shelf mmWave radar kit which consists of a TI IWR1843BOOST sensor boardoperating at 77-81GHz and a TI DCA1000EVM data capture board. The IWR1843BOOST board has 58-degree and 15-degree angular resolution in elevation and azimuth, respectively. It is placed upright to configure a 15-degree angular resolution in elevation for face localization. TI mmWave studio version 02.01.01 is used for radar processing and PyTorch version 1.4.0 for model training on a Windows PC with an AMD Ryzen 73700X CPU and a NVIDIA GeForce RTX 3090 GPU. Table 1 shows the radar configuration in detail. In particular, fully enabling 3x4 MIMO on the IWR1843BOOST board, the chirps per frame generated are limited up to 32. Based on the default radar configuration in TI mmWave studio, the frame periodicity of 10ms (i.e., frame rate of 100Hz) is used. Since the frame periodicity limits the ramping time, the frequency slop of 100.0 MHz / us is set to maximize the radar bandwidth of 3.6GHz. In addition, a coordinate calibration method is applied to align both mmWave and image data configuration (e.g., timeline and displacement between hardware).

[0233] In real-world deployment, the first 3s radar data is used for dual-locating, hence no additional privacy concern is raised by biometric verification. The prototype is powered using a 5V power jack and connected to the PC for real-time processing. The face localization takes 400ms on average for processing including range localization of 180ms, biometric verification of 10ms, and face-matching of 210ms. A 500ms sliding window is employed with a 3s time window to feed data to the mmWave FER model, and the latency of each model inference is 4.5ms or 13ms on average in the GPU or CPU-enabled mode, respectively. The psutil version 5.9.0 is employed to track resource usage. In watching a 10mins video, the CPU utilization of dual locating is 6.2% on average, and the memory footprint is up to 368MB (the usage may vary on different platforms). For model inference, the CPU utilization is 49.8% on average and the memory footprint is up to 610MB in the CPU-enabled mode, while the GPU utilization is 44.5% on average and the peak memory footprint reaches 3890MB in the GPU-enabled mode. In practice, the prototype can be placed on top / bottom of or next to the multimedia device, as shown in Figures 20B to 20F. In addition, the prototype can be integrated with different embedded platforms (e.g., Raspberry Pi or NVIDIA Jetson series) for portable deployment. Evaluation Setup

[0234] A range of multimedia IoT devices were used which provide different viewing distances, as shown in Table 2.

[0235] Various scenarios were set-up in a living room (2.9mx4.2m) of a residential apartment. 10 subjects (3 females and 7 males, age ranging from 23 to 32, height ranging from 156cm to 189cm) were recruited. mmWave and video datasets were collected with 7 standard facial expressions ("neutral", "happiness", "sadness", "surprise", "fear", "disgust", and "anger"). A depth camera (i.e., Zed 2 Camera operates at 30fps) was used to collect videos. During data collection, subjects performed different postures (e.g., sit on sofa, sit on the ground, and stand) and body motions (e.g., playing games, typing, and leg shaking) at a subject-to radar distance ranging from 0.3m to 3m. Subjects’ faces were not strictly facing the radar as this may affect facial expressions. Subjects perform these activities in a natural way, hence a slight variation (e.g., less than 10-degree) may be possible. Pre-trained ResNet18 based on the FER Plus expression dataset was used, and fine-tuned on the video dataset collected from the camera as the ground truth. Dual-locating Performance

[0236] The performance of the dual-locating approach and its impact on face localization was evaluated. This employed vision based face detection in to locate face and output angle ranges as the ground truth. The root mean squared error (RMSE) divided by the vision angle range as metric of face localization error drift, defined as:

[0237] where ^^^^^^ ^^and ^^^^^^denote the center of facial angle range denotes the ground truth angle range located by dual-locating and vision, respectively, and ^^ ^^^^denotes the ground truth angle range. Impact of Different Distances.

[0238] Facial angle range may inversely vary as the subject-to-radar distance increases, e.g., a far distance results in a relatively small facial angle range. In this experiment, the impact of different distances on face localization in dual-locating is evaluated. Three subjects sat at seven different viewing distances from a 27-inch screen (i.e., the same experimental setup as in Figure 20D in turn and perform facial expressions following a recorded video. Figure 21A demonstrates that the distance increase yields a slight increase of error drift on face localization, e.g., the error drift rate is settled in the range from 5.7% at 0.3m to 18.9% at 3m. It is also observed a little increase of error drift as distance increases. This may be due to a small angle range offset added in the process of face-matching. Moreover, due to physical variation, the standard deviation of error drift (e.g., red error bar) has a minor increase as distance increases (e.g., from 1.7% at 0.3m to 4.8% at 3m). Thus, the results indicate that dual- locating can effectively enable face localization at different subject-to-radar distances with minor error drift. Multi-target Detection

[0239] Next the performance of dual-locating for locating multiple faces was evaluated using a scenario of watching TV with two subjects. Due to the upright radar setup, the azimuth FOV is limited, so one subject (S1) sits on the ground 1m away and another subject (S2) stands 2m away, as shown in Figure 20F. Figure 21B shows that both subjects’ faces can be successfully located in angle range heatmap when performing "surprise" facial expressions simultaneously. The results show that dual-locating can locate multiple subjects with accurate range bins and facial angle ranges marked by highlighted blocks. Also, compared to the ground truth of facial angle range, dual-locating achieves low error drift for both subjects, e.g., 3.3% for S1 and 8.3% for S2. In addition, the visual results present that the heat face area (i.e., spatial facial information) of S2 is relatively smaller than that of S1 due to the impact of subject-to-radar distance. Thus, dual-locating can locate face accurately for multiple targets. Cross-transfer Performance

[0240] The performance of the cross transfer pipeline is evaluated with the respect to performance of feature alignment and loss optimization compared to the state-of the-arts. For a fair comparison, both small-scale image and mmWave datasets were selected in a scenario of watching iPad to avoid the impact of viewing distance, as shown in Table 2. 40 samples perexpression per subject, and each sample is recorded in a 3s time window by both camera and mmWave radar.

[0241] The following cross-domain learning approaches were used as baselines. All of approaches share with the same pre-trained ResNet18. • KD a conventional cross-domain learning approach that uses knowledge distillation techniques without latent feature alignment support as described in Lin Wang and Kuk- Jin Yoon. 2021. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021). • Keypoint, a facial landmark based image-to-sensing transformation approach described in Xingzhe Song, Kai Huang, and Wei Gao.2022. FaceListener: Recognizing Human Facial Expressions via Acoustic Sensing on Commodity Headphones (IPSN’22), which uses Mediapipe described in Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al.2019. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172 (2019).

[0037] to generate facial landmarks. • S-cross is a standard unsupervised cross learning approach that leverages on a distance- based loss function to train a student model. It uses a vanilla autoencoder for latent feature alignment as described in Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Throughwall human pose estimation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.7356–7365. • Image described in Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang. 2016. Training deep networks for facial expression recognition with crowd- sourced label distribution. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. 279–283., which uses the fine-tuned image model as the ground truth. Feature Alignment Performance.

[0242] To quantify the effectiveness of the proposed feature alignment mechanism in the cross- transfer pipeline, the mean squared error (MSE) is employed as metric to measure the ED loss descent (i.e., distance-based similarity) between the latent features of image and mmWave models. As shown in Figure 22A, mmFER according to the above described techniques remains the lowest MSE loss during training and achieves 0.39 at 50-epoch, outperforming both KD (2.73) and S-cross (0.61). Since two separate CNN blocks are used to safely explore key latent features without information loss in the proposed mechanism, mmFER successfully achieves a lower MSE loss than S-cross that uses a vanilla autoencoder. The results demonstrate that mmFER outperforms both KD and S-cross in achieving the highest model accuracy of 86.36%, indicating that a lower MSE loss enabled by the cross-transfer pipeline leads to higher model accuracy. Loss Optimization Performance.

[0243] In this experiment, the performance of the proposed hybrid learning loss function was compared with baselines. This followed a widely-used clustering benchmark, and employed the silhouette score (i.e., ranging from1 to 1 and being better if closing to 1) measured by both intra-class and inter-class distances 2. The silhouette score of Image as the ground truth was also computed. Figure 22B shows that mmFER achieves the best silhouette score of 0.52 on average, outperforming both KD (0.12) and S-cross (0.21). In particular, mmFER achieves the highest silhouette scores across all expressions over both baselines.

[0244] Besides, the silhouette scores achieved by mmFER are closer to the ground truth, implying that the proposed hybrid learning loss can reduce intra-class distance and enlarge inter-class distance for improving loss optimization in training image to mmWave.

[0245] Moreover, the receiver operating characteristic (ROC) curve can be used to compare the training performance of mmFER with the baselines. Figure 23A shows that mmFER achieves the highest AUC (i.e., area under the curve) of 97.17 and the best training model accuracy of 86.36% on average, outperforming the baselines. This experiment demonstrates that cross-transfer pipeline can notably enhance model performance and transferability in cross-domain. mmFER Performance

[0246] The overall performance of mmFER from different aspects was evaluated.

[0247] The FER accuracy of mmFER was first evaluated. This involved training an mmWave FER model based on all subjects’ dataset with user-dependent setting. The dataset is randomly split to 80% for training and 20% for testing by 3 rounds and train a model per round. Figure 23B shows that mmFER achieves a FER accuracy of 86.19% on average. The accuracy for each subject falls into the range between 81.43% to 93.00% with a small standard deviation of 3.74% on average.

[0248] The confusion matrix for each facial expression was also calculated and as shown in Figure 24A, the FER accuracy of mmFER for all expressions is settled in the range between 83.0% to 89.1% with a small standard deviation of 2.10%. In particular, "surprise" expression achieves the highest accuracy among all expressions. The experiment demonstrates that mmFER is effective to recognize different subjects’ facial expressions with high accuracy.

[0249] For a user independent mode, the physical variations among different subjects may challenge the generalization performance of mmWave FER model especially for unseen subjects. The generalization of mmFER in user-independent condition was evaluated. To quantify variations, three subjects are selected and their facial mmWave data distribution visualized by each expression using t-SNE shown in Figures 25A to 25F. The results demonstrate that different subjects lead to notable data pattern variations even for the same expression.

[0250] Applying the leave-one-subject-out validation (i.e., leaving one subject independent as unseen and training model based on the remaining subjects’ dataset) 10 user independent models were trained and tested using 10-fold cross-validation. Figure 24B shows that the FER accuracy by expressions drops, but it still remains at 71.70% on average with a standard deviation of 6.54%. In particular, the accuracy of "surprise" expression is decreased by 22.1%, implying that the variations may be enlarged due to the intrinsic large range of facial muscle movements (e.g., data distribution is more scattered as shown in Figure 25B). This experiment demonstrates that mmFER has good generalization performance to recognize unseen subjects’ facial expressions.

[0251] The impact of detection distance was investigated. The subject-to-radar distance may impact on spatial facial information extraction (i.e., a far distance may yield a small heat face area shown in Fig. 13(b)). In this experiment, the impact of distances on FER accuracy is evaluated using the experimental setup described above and testing mmFER in a range between 0.3m to 3m to cover use scenarios in Table 2.

[0252] Figure 26B plots that the FER accuracy decreases from 86.18% at 0.3m to 64.5% at 3m on average with a standard deviation of 3.65%. Since the angular resolution of mmFER is limited and the actual facial angle range becomes smaller as distance increases, the spatial facial information extracted from mmWave signals may be reduced, resulting in accuracy drop. However, the results prove that mmFER can still effectively achieve a good FER accuracy of 80.57% on average within 2.5m, which can be applied in a wide range of multimedia IoT applications.

[0253] The impact of face orientation (i.e., the angle of head towards the radar) on FER accuracy was evaluated as subject-to-radar distance increases. 4 different angles of head in azimuth: 0-degree (i.e., facing directly to the radar), 30-degree, 60-degree, and 90-degree were used. Subjects turned their faces left or right randomly. Comparing the performance of mmFER with the image model, Figure 26A plots the FER accuracy heatmaps with different orientations and distances using image and mmWave models. It is observed that the accuracy of the image model at 0-degree is decreased from 96.64% to 83.3% as distance increases, and mmFER achieves a similar result. The results also reveal that the accuracy drops for both models as angle of head increases at each distance, e.g., down to 53.80% (image) and 46.94% (mmWave) from 0-degree to 30-degree. In particular, both models fail when angle of head is larger than 30-degree.

[0254] The impact of body motions was evaluated by ask three subjects to perform different body motions (e.g., playing games, typing, and leg shaking) in turn when watching videos (e.g., 10 rounds each subject) using the experimental setup shown in Figure 20D. Figure 27A 19(a) shows that mmFER achieves an accuracy of 82.2% on average when performing these body motions. In particular, the result of leg shaking has a slightly lower accuracy of 80.06% compared to that of playing games and typing, implying that a larger range of body motionshas more impact on face localization. Overall, the results prove that mmFER can avoid body motions to ensure system robustness.

[0255] The impact of different postures was evaluated using the same experimental setup, with the three subjects performing three regular body postures (i.e., sit on chair, sit on ground, and stand). Figure 27B shows the accuracy with three postures for three subjects. The result shows that mmFER achieves an accuracy of 81.15% on average with a standard deviation of 2.59%. It is observed that the accuracy of both Stand and Sit on Ground drops slightly to 78.33% and 80.75%, respectively. This implies that spatial facial information extracted from mmWave signals may be affected by different postures due to the change of face orientation in elevation. Overall, this experiment demonstrates that mmFER is robust and resilient to a range of postures.

[0256] The impact of wearable accessories on head or face is common in reality, different accessories may impact on FER performance due to materials (e.g., plastic or metal frames) and occlusions (e.g., partial or full face occlusions). This was investigated in a scenario of watching iPad in Table 2, with the subject wearing regular accessories in five different ways (e.g., mask, glasses, cap, 3D glasses, and 3D glasses & mask) when reacting videos for 10 rounds. Figure 28 shows that mmFER successfully achieves a FER accuracy of 80.47% on average with a standard deviation of 2.8% by wearing accessories. Especially, wearing a cap has less impact with an accuracy of 83.86% on average, while the accuracy of wearing glasses (81.52% on average) and 3D glasses (80% on average) slightly drops as the frame materials may affect the reflection of mmWave signals. In addition, wearing mask yields an accuracy decrease, but mmFER still achieves an accuracy of 80% on average due to fair penetration. Even for wearing 3D glasses & mask, mmFER achieves a good accuracy of 77.07% on average. It implies that mmFER can work in a scenario with wearing mask regulations. Thus, the results indicate that mmFER is robust for wearing accessories.

[0257] Sensor Configuration In the above examples, the IWR1843BOOST board is placed upright, resulting in 58-degree of azimuth angular resolution and 30-degree of FOV. This setup may limit multi-target detection in azimuth in reality. Since the dual-locating approach is designed to locate multiple targets in azimuth for any angle resolution and FOV, this situationcan be improved by using an mmWave radar with wider FOV or deploying multiple radars, for example to provide a 140-degree FOV.

[0258] In principle, the detection range of mmFER is subject to angular resolution which is limited by the number of RX antennas. Although a high-resolution AoA estimation can enhance angular resolution, the accuracy of mmFER may still drop when detection range increases, and so angular resolution can be improved for a longer detection range, for example using a TI MMWCAS-RF-EVM mmWave radar that provides an angular resolution of 1.4degree by 86 virtual RX antennas in azimuth and 18-degree by 4 virtual RX antennas, and further evaluate mmFER. Different Subjects

[0259] Experiments show that cross-transfer pipeline significantly improves the performance of mmFER based on small-scale mmWave datasets. According to Figure 25 different subjects may generate a large data distribution shift caused by individual face and expression variation, hence the performance of mmFER may vary with the diversity and scale of dataset.

[0260] In as further experiment, the performance of face localization in dual-locating with respect to physical variations among subjects (e.g., height, face shape, and facial muscle movement) was evaluated. To achieve this, three subjects were positioned 1m away from a 27- inch screen and perform facial expressions following a recorded video, using an experimental setup that was otherwise similar to above experiments. Figure 29 shows the cumulative distribution function (CDF) of error drift when locating different faces. It is observed that error drift of face localization exists among subjects, but the error drift rate is below 16.7% (i.e., less than 2.2-degree error for 80% probability). The error drift differences among subjects are minor, e.g., 12.72% (S1), 12.28% (S2), and 13.23% (S3). This experiment demonstrates the robustness of locating face with low error drift. Illumination Impact

[0261] To enhance the user experience, watching in low illumination environment (i.e., weak ambient light) is quite common in reality. Accordingly, it is important to determine whether mmFER is able to recognize facial expressions in dark (i.e., turn off room lights) compared to a well illuminated scenario. To test this, an experiment was performed by positioning a subject1m away from a TV, and employ ROC curve to present model classification performance in detail. Figure 30 demonstrates that the FER accuracy of Image dramatically decreases from 94.64% in normal light on to 20% in dark, while mmFER successfully remains the accuracy changes at the same level (e.g., from 85.60% in normal to 83.52% in dark). The results prove that vision-based FER system may easily fail in low illumination environment. Due to the illumination-free capability of mmWave sensing, mmFER can effectively achieve high system resilience in various illumination conditions. CONCLUSION

[0262] Accordingly, an mmWave radar based FER can be provided that detects subtle facial muscle movements caused by facial expressions for facial expression recognition. mmFER is a device-free solution that enables robust FER and delivers valuable assessment of users’ engagement and reactions to multimedia IoT applications. Extensive evaluations demonstrate that mmFER is resilient to various real-world settings with high accuracy, hence it is capable of deployment in a wide range of scenarios with a flexible detection range for multiple users.

[0263] Throughout this specification and claims which follow, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” or “comprising”, will be understood to imply the inclusion of a stated integer or group of integers or steps but not the exclusion of any other integer or group of integers. As used herein and unless otherwise stated, the term "approximately" means ±20%.

[0264] Persons skilled in the art will appreciate that numerous variations and modifications will become apparent. All such variations and modifications which become apparent to persons skilled in the art, should be considered to fall within the spirit and scope that the invention broadly appearing before described.

Claims

THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS: 1) A facial expression sensing method, the method including, in one or more processing devices: a) acquiring radar data from a radar sensor configured to perform range sensing in an environment including a person; b) analysing the radar data to determine facial radar data indicative of radar data captured from a face of the person; c) determining a facial expression by using facial radar data and at least one computational model, the at least one computational model being at least partially indicative of relationships between facial radar data and different facial expressions; and, d) generating an indication of the facial expression. 2) The method of claim 1, wherein the method includes, in the one or more processing devices: a) analysing the radar data to perform localisation of the person within the environment; and, b) determining the facial radar data based on the localisation. 3) The method of claim 2, wherein the method includes, in the one or more processing devices, performing localisation of the person at least in part by analysing the radar data to identify changes indicative of a biometric. 4) The method of claim 3, wherein the biometric includes at least one of: a) a heat beat; and, b) respiration. 5) The method of any one of the claims 1 to 4, wherein the method includes, in the one or more processing devices: a) analysing changes in the radar data to identify moving objects; b) filtering the radar data to remove moving objects and thereby identify candidate static objects; and, c) localising the person using the candidate static objects. 6) The method of claim 5, wherein the method includes, in the one or more processing devices, identifying moving objects at least one of: a) using an adaptive velocity threshold; and, b) using changes in range profile.7) The method of claim 5 or claim 6, wherein the method includes, in the one or more processing devices: a) determining a range bin for each candidate object; and, b) localising the person at least in part based on the range bins of the candidate objects. 8) The method of claim 7, wherein the method includes, in the one or more processing devices, filtering the range bin to remove ambient noise. 9) The method of claim 8, wherein the method includes, in the one or more processing devices, analysing the radar data to remove ambient noise and determine range bins using an adaptive cell-averaging constant false alarm rate (CFAR) algorithm. 10) The method of claim 8 or claim 9, wherein the method includes, in the one or more processing devices, analysing the filtered range bins to identify changes indicative of a biometric. 11) The method of any one of the claims 1 to 10, wherein the radar sensor includes a 2D array of sensing elements that captures radar data over azimuthal and elevational directions and wherein the method includes, in the one or more processing devices: a) localising the range and azimuthal location of the person; b) analysing the radar data for the azimuthal location to determine an elevational angle range of a face of the person; and, c) determining the facial radar data using the elevational angle range. 12) The method of claim 11, wherein the method includes, in the one or more processing devices, determining the elevational angle by: a) performing high-resolution frequency-wavenumber spectrum analysis to generate enhanced angle range heatmaps; and, b) using a Gaussian Mixture Model (GMM) to determine the elevational angle range of the face. 13) The method of any one of the claims 1 to 12, wherein the method includes, in the one or more processing devices: a) determining one or more features from the facial radar data; and, b) applying the one or more features to the computational model. 14) The method of claim 13, wherein the method includes, in the one or more processing devices, determining the one or more features using changes in the facial radar data.15) The method of claim 13 or claim 14, wherein the one or more features include at least one of: a) heatmaps indicative of changes in facial radar data, the changes in facial radar data being indicative of facial muscle movements; and, b) Doppler heatmaps. 16) The method of any one of the claims to 1 to 15, wherein the at least one computational model is obtained at least in part by applying machine learning to reference facial radar data acquired from persons having different known facial expressions. 17) The method of claim 16, wherein the at least one computational model is obtained by applying machine learning to heatmaps indicative of changes in the reference facial radar data. 18) The method of claim 16 or claim 17, wherein the at least one computational model is obtained at least in part by applying machine learning to reference image data acquired from persons having different known facial expressions. 19) The method of any one of the claims 16 to 18, wherein the at least one computational model is obtained by cross domain training using a combination of reference image data and reference facial radar data. 20) The method of any one of the claims 16 to 19, wherein the at least one computational model is obtained by: a) applying reference facial radar data to at least one radar data computation model to determine latent radar features; b) applying reference image data to at least one image data computation model to determine latent image features; c) using an autoencoder to align the latent radar features with the latent image features; and, d) using the aligned range features and a pre-trained image classifier model to train the at least one computational model. 21) The method of claim 20, wherein the at least one computational model is obtained using a hybrid loss function that fuses a plurality of loss functions to avoid overfitting. 22) The method of claim 20 or claim 21, wherein the at least one computational model is obtained by:a) applying azimuthal reference facial radar data to a first model to extract one or more first features; b) applying elevational reference facial radar data to a second model to extract one or more second features; and, c) combining the first and second features to generate the latent radar features. 23) The method of any one of the claims 1 to 22, wherein the method includes, in the one or more processing devices: a) identifying content being viewed by the person; and, b) associating an indication of the content and the facial expression. 24) The method of claim 23, wherein the method includes, in the one or more processing devices: a) transmitting an indication of the content and the facial expression; and, b) deleting at least one of: i) radar data; and, ii) facial radar data. 25) The method of any one of the claims 1 to 24, wherein the radar sensor is at least one of: a) a radar; b) a mmwave radar; and, c) a lidar. 26) A facial expression sensing apparatus including: a) a radar sensor configured to perform range sensing in an environment including a person; and, b) one or more processing devices configured to: i) acquiring radar data from the radar sensor; ii) analysing the radar data to determine facial radar data indicative of radar data captured from a face of the person; iii) determining a facial expression by using facial radar data and at least one computational model, the at least one computational model being at least partially indicative of relationships between facial radar data and different facial expressions; and, iv) generating an indication of the facial expression.27) A method of training at least one computational model for use in facial expression sensing, the method including, in one or more processing devices: a) acquiring reference facial radar data from persons having different known facial expressions; b) acquiring reference image data from persons having different known facial expressions; and, c) obtaining the at least one computational model by cross domain training using a combination of reference image data and reference facial radar data. 28) A method according to claim 27, wherein the method includes: a) acquiring radar data from a radar sensor configured to perform range sensing in an environment including a person; and, b) analysing the radar data to determine the facial radar data, the facial radar data being indicative of radar data captured from a face of the person. 29) The method of claim 27 or claim 28, wherein the method includes, in the one or more processing devices: a) applying reference facial radar data to at least one radar data computation model to determine latent radar features; b) applying reference image data to at least one image data computation model to determine latent image features; c) using an autoencoder to align the latent radar features with the latent image features; and, d) using the aligned range features and a pre-trained image classifier model to train the at least one computational model. 30) The method of claim 29, wherein the method includes in the one or more processing devices, using a hybrid loss function that fuses a plurality of loss functions to avoid overfitting. 31) The method of claim 29 or claim 30, wherein the method includes, in the one or more processing devices: a) applying azimuthal reference facial radar data to a first model to extract one or more first features; b) applying elevational reference facial radar data to a second model to extract one or more second features; and,c) combining the first and second features to generate the latent radar features. 32) A system for training at least one computational model for use in facial expression sensing, the system including one or more processing devices configured to: a) acquire reference facial radar data from persons having different known facial expressions; b) acquire reference image data from persons having different known facial expressions; and, c) obtain the at least one computational model by cross domain training using a combination of reference image data and reference facial radar data.

Images