Method for detecting emotion using radio frequency

Abstract

The present invention relates to a method for detecting an emotion in a subject (11) comprising the following steps: - emission of a first radio frequency signal (4) towards the subject, - reception of a second radio frequency signal (5) containing a physiological signal due to the reflection of said first radio frequency signal on the subject, - digital processing (6) of the second radio frequency signal (5) so as to generate respiratory data (6a) and / or cardiac data (6b), - application of several time windows F1 of duration T1 in a sliding manner on the respiratory data and / or cardiac data, - application of several time windows F3 of duration T3 in a sliding manner on the respiratory data and / or cardiac data;duration T3 being less than duration T1, - prediction of an emotion by an emotion model (12) based on artificial intelligence from data from windows F1 and data from windows F3. See figure 3;

Description

Title of the invention: Method for detecting an emotion by radio frequency.

[0001] The present invention relates to a method for detecting an emotion in a subject using an artificial intelligence-based emotion model. The invention also relates to a system comprising a processing unit for implementing such a method.

[0002]

[0003] The invention finds a particularly interesting application in the detection of a user's emotion at a workstation. However, the invention has a broader scope since it can be applied to many other professional or personal fields such as human-computer interaction, virtual reality, games, and healthcare.

[0004]

[0005] Society continues to evolve towards an increasingly present integration of technology into our daily lives.

[0006] Our relationship to consumption, health, education, information, and even the very functioning of our society now requires ever-greater mastery of the digital world. It must be said that the technological advances of recent decades and the advent of the Internet and a more globalized economy have greatly facilitated access to and use of high-tech products, from both a financial and functional point of view.

[0007] And yet, digital interfaces remain largely passive with respect to their users and what motivates them: their intentions, their emotions beyond their interactions. They are incapable of recognizing episodes of frustration when a user is confronted with a poorly designed web interface. It is impossible to monitor the concentration and fatigue of an operator at a hazardous industrial workstation. It is impossible to recognize the despair of a student overwhelmed by knowledge in their learning, or conversely, to celebrate a moment of happiness with a gamer proud of their prowess in a video game. If these interfaces were capable of sensing the emotions and intentions of their users, we would no longer be obliged to use explicit commands and would benefit from a more natural and therefore even more accessible interaction.Furthermore, machines would demonstrate their ability to assist humans in regulating their own emotions, resulting in greater safety and well-being.

[0009] Today, most approaches in affective computing rely on the use of audiovisual information such as facial expressions, posture, or prosody. These are very useful because they are easily accessible, but they are not without potential biases. Since facial expressions are a mode of social communication, they are rarely expressed by a user alone in front of their screen. Conversely, they may be constrained to conceal an emotion of opposite valence. The same is true for the expression of emotions through the voice. Moreover, facial expressions are not necessarily visible within the camera's field of view, and their detection may be limited in an uncontrolled environment due to lighting conditions.

[0010]

[0011] To overcome these limitations, one can focus on more physiological, and therefore more unconscious, signals induced by the regulation of the autonomic nervous system, which is closely linked to emotional changes. Systems using an electrocardiogram on the chest to monitor cardiac activity are known. Electrodes distributed on the body can measure galvanic conductance and thus variations in perspiration. An electroencephalogram can characterize neuronal activity, while a pulse oximeter can monitor a user's pulse. By processing these signals with trained artificial intelligence modules, it is possible to extract a certain dynamic of the subject's state and feelings. Such measurement solutions, practical in the laboratory, quickly prove impractical in everyday environments.

[0012] Non-invasive sensors used in emotion recognition are known. The use of thermal cameras to capture surface temperature differences on the face is becoming increasingly popular. While this approach has the advantage of focusing on the physiological component of emotion, it nevertheless has the drawback of requiring the target's face to be visible.

[0013]

[0014] In recent years, some researchers have also used radar technologies on certain radio frequency ranges to remotely detect micromovements of the rib cage and, indirectly, of the heart muscles. From the processing of these bodily micromovements, they have accessed real-time changes in heart rhythm that contain diagnostic information for assessing emotions.

[0015] US patent US20170311901A1 is known, describing a radar system for emotion recognition. The radar used is a frequency-modulated continuous wave (FMCW) radar. Emotion characterization is achieved through a discrete classification of four emotions. over a 2-minute time window.

[0016] We know the document Siddiqui HUR et al., (2021), “Respiration Based Non-Invasive Approach for Emotion Recognition Using Impulse Radio Ultra Wide Band Radar and Machine Leaming.”, Sensors, 21(24), Article 24. https: / / doi.org / 10.3390 / s21248336, describing a method for extracting the number of breaths per minute from a signal from a UWB radar.

[0017] Khan et al., (2021), “Deep learning framework for subject-independent emotion detection using wireless signals,” PLOS ONE, 16(2), e0242946. https: / / doi.org / 10.1371 / journal.pone.0242946, are known to describe a radar system for emotion recognition. Emotion characterization is performed by discrete classification of several emotions over long time windows (2 min). The paper describes an artificial intelligence model built directly from the raw radar signal data.

[0018]

[0019] We are familiar with the paper by Dang et al., (2022), “Emotion recognition method using millimeter wave radar based on deep leaming.”, IET Radar, Sonar & Navigation, 16(11), 1796-1808. https: / / doi.org / 10.1049 / rsn2.12297, describing a 77 GHz FMCW millimeter wave radar for emotion recognition. This paper describes signal processing to extract cardiac and respiratory activity, both of which are taken into account in the model and combined for classification.

[0020]

[0021] The present invention aims at a new real-time method of emotion recognition using radar.

[0022] Another objective of the invention is accurate and reliable recognition of emotions.

[0023] At least one of the objectives is achieved with a method for detecting an emotion in a subject including the following stages:

[0024] - emission of a first radio frequency signal towards the subject,

[0025] - reception of a second radio frequency signal containing a physiological signal due to the reflection of said first radio frequency signal on the subject,

[0026] - digital processing of the second radio frequency signal so as to generate respiratory and / or cardiac data,

[0027] - application of several time windows Fl of duration Tl in a sliding manner based on respiratory and / or cardiac data,

[0028] - application of several time windows F3 of duration T3 in a sliding manner on respiratory and / or cardiac data; the T3 duration being less than the T1 duration, and

[0029] - prediction of an emotion by an artificial intelligence-based emotion model based on data from windows Fl and data from windows F3.

[0030]

[0031] The method according to the invention offers a non-invasive solution, usable in a natural environment and capable of recognizing the emotional dynamics of a subject as close as possible to their physiology without requiring the wearing of accessories in contact with the body.

[0032] By using large and small windows, the characterization of cardiac data and the extraction of components of interest to feed the associated AI model are improved.

[0033] The analysis of large windows makes it possible to consider the overall physiological activity throughout the user's observation period.

[0034] Small window analysis makes it possible to consider more local activities specific to an emotional event and to characterize in the signal from physiological activity specific patterns of more local emotional events to be detected.

[0035] The time windows Fl preferably have the same fixed duration, but it is possible to consider time windows Fl with different durations. The same applies to the time windows F3.

[0036] Performing analysis of physiological signals over short time windows makes it possible to obtain predictions over short durations, which allows for real-time prediction of sudden emotional events during the exploitation phase of the emotion model.

[0037] It is thus possible to evaluate the dynamics of emotions by targeting the intensity of one or more specific emotions or internal states or cognitive intentions over time on a human or animal subject.

[0038] Physiological data include, for example, but not limited to, fluctuations in heart rate, respiratory fluctuations, eye activity, galvanic skin responses and blood pulses.

[0039]

[0040] With the process according to the invention, two dynamics of emotional physiology are considered: a slow dynamic corresponding to progressive variations of the targeted emotional dimension, such as stress or joy, and a much faster dynamic, in the space of a few seconds or less, corresponding to more sudden and local variations of this same emotional dimension.

[0041] It is considered that information specific to the experienced emotion occurs in physiological activity over latencies much shorter than 1 minute, within a few seconds. It is this information that is sought by using short time windows as labeled data in addition to long windows, the latter allowing, for example, the extraction of a reference emotional level.

[0043] According to an advantageous feature of the invention, the data from the time window Fl can be directly transmitted to the emotion model and / or can undergo calculation of a value of at least one respiratory and / or cardiac biomarker, this value then being transmitted to the emotion model.

[0044]

[0045] In particular, the calculation of a value for at least one respiratory and / or cardiac biomarker may include at least one of the following calculations:

[0046] - calculation of heart rate variability (HRV for “heart rate va riability”),

[0047] - calculation of respiratory rate variability (RRV) or of a respiratory pattern (BPA for “Breathing Pattern Analysis”) or of a respiratory amplitude and its variability (RAV for “Respiratory Amplitude and Variability”),

[0048] - calculation of a respiratory sinus arrhythmia (RSA for “Respiratory Sinus”) Arrythmia” in English).

[0049] The RSA is representative of the interaction between cardiac activity and respiratory activity, reflecting the activity of the parasympathetic autonomic nervous system.

[0050] Thus, the method according to the invention can be applied to cardiac activity alone, to respiratory activity alone, or to both activities.

[0051] Depending on the activity used, one or more of the functions listed above can be calculated.

[0052]

[0053] According to an advantageous feature of the invention, the duration Tl is a minimum of 30 seconds with a sliding step of a minimum of 1 millisecond; preferably the duration Tl is between 30 seconds and 5 minutes with a sliding step between 1 millisecond and 15 seconds; ideally the duration Tl is between 30 seconds and 1 minute 15 seconds with a sliding step between 3 seconds and 5 seconds.

[0054]

[0055] The duration Tl corresponds to the large time windows which are applied in a sliding manner, with a fixed time offset or not.

[0056] The physiological data obtained from these long windows, in particular beyond one minute, allow for example the calculation of heart rate variability (HRV) which requires waiting for several heartbeats to be calculated.

[0057]

[0058] Advantageously, the duration T3 can be a minimum of 375 milliseconds, or between 375 milliseconds and 30 seconds, or between 1 and 5 seconds, with a minimum step of 1 millisecond.

[0059] The long window can have a duration of 1 min (Tl), and be applied in a sliding manner in 3 second steps throughout the entire observation period of the subject.

[0060] The short window can have a duration of 3 seconds (T3), and be applied in a sliding manner in steps of 500ms during the entire duration of observation of the subject.

[0061]

[0062] According to the invention, the first radio frequency signal can be a continuous wave radar signal modulated according to an FMCW protocol (for “Frequency-Modulated Continuous Wave” in English) or an ultra-wideband ULB signal (or UWB for “Ultra Wide Band” in English)

[0063] By way of example, the FMCW radar can operate in both frequency bands [57-64 GHz] and [76-81 GHz] up to a maximum of 4 GHz. The UWB radar can operate in a range between 6 and 8.5 GHz.

[0064]

[0065] According to the invention, respiratory and / or cardiac data can undergo a normalization step before the application of time windows.

[0066]

[0067] Thus, according to the invention, data from cardiac and / or respiratory activities are processed independently through two long sliding windows Fl and short sliding windows F3 to extract one or more data points from among: - All cardiac signals present in successive windows Fl - All respiratory signals present in successive windows Fl - Key cardiac measurements derived from Fl timescales: for example HRV (“Heart Rate Variability”) - Key respiratory measures derived from Fl timescales: for example RRV (“Respiratory Rate Variability”), BPA (“Breathing Pattern Analysis”), RAV (“Respiratory Amplitude Variability”) - Measures reflecting the synchronization between cardiac and respiratory activity derived from Fl temporalities, for example RSA (“Respiratory Sinus Arrythmia”) - All cardiac signals present in successive F3 windows - All respiratory signals present in successive windows F3.

[0068]

[0069] This data is then used by the emotion model to predict emotions.

[0070] Additional data from other types of sensors than the aforementioned antennas can be used in operation to help weight the output results of the chosen Deep-Leaming AI model.

[0071] With the invention, through the analysis of the characteristics of the data in the time and frequency domains, the periodic heart rate and respiratory data are extracted.

[0072] The outputs of the radar sensors are representations of physiological activities in their own dimensional, temporal, or frequency spaces integrated into the noise signal. The signal is filtered and processed to reveal the physiological signal from the converted baseband signal as accurately as technically possible.

[0073] A clutter suppression algorithm can be implemented. Through a (time-sliding) transmission and reception window applied to the desired signature, static states as well as slow movements, which differ from the desired modulations—such as rib cage activity, cardiac and respiratory activity—are excluded. The interference between the harmonics and the fundamental frequency of the heart and respiration can be improved by implementing an empirical decomposition algorithm into ensemble modes and decomposing the signal into an Intrinsic Mode Function (IMF). The improvement of the cardiac and respiratory rhythm can be achieved through the continuous wavelet transform (CWT).

[0074] The invention makes it possible to extract the complete cardiac and respiratory signals according to the short and long time windows chosen, but also classic characteristics of cardiac and respiratory activities such as HRV, RRV, BPA, RAV, RSA and other characteristics within the same heartbeat or in relation to the relationship between different cardiac characteristics of successive heartbeats.

[0075] The data are then optionally normalized and missing values ​​can be filled in if necessary. All this data can then optionally be converted into vectors for use as inputs in artificial intelligence-based emotion models.

[0076] According to an advantageous feature of the invention, the emotion model consists of one or more learning models, simple, deep learning or preferably hybrid.

[0077] The emotion model may include a hybrid recurrent neural network. The hybrid recurrent neural network allows the artificial intelligence model to be trained and evaluated to detect and classify significant changes in cardiac and respiratory activities based on calculated cardiac and respiratory characteristics.

[0078] The emotion model may include a fusion between a gated recurrent unit (GRU) and a long- and short-term memory network (LSTM). Preferably, The hybrid recurrent neural network results from such a fusion.

[0079] Such a fusion is well suited to predictive analysis and classification of time series on a reasonably small dataset. This fusion results in an improved and more accurate model. The GRU and LSTM networks are combined to leverage their respective strengths, notably the GRU's ability to identify short-term dependencies and the LSTM's ability to identify long-term dependencies. By combining the LSTM and GRU models, the invention identifies different types of temporal patterns or dependencies present in the data, thereby improving the emotion model's ability to make accurate predictions while preserving robustness and generalizability. This also reduces the risk of overfitting and makes the model more resilient to noise or uncertainties in the data.

[0080] Moreover, the simplicity of the GRU improves the explainability of algorithmic decisions.

[0081] Advantageously, the emotion model may include:

[0082] - a slow dynamics model dedicated to data from temporary windows Porelles Fl,

[0083] - a fast dynamics model dedicated to F3 time window data, and

[0084] - a decision-making algorithm common to the slow dynamics model and to rapid dynamics model.

[0085]

[0086] According to an advantageous feature of the invention, for example in the context of using an FMCW radar, the transmission of the first radio frequency signal can be carried out by means of several transmitting antennas and the reception of the second radio frequency signal can be carried out by means of several receiving antennas according to a MIMO (for "Multiple-Input Multiple-Output" technology).

[0087] The FMCW radar can therefore be based on a coherent MIMO system with more than one transmitting antenna and more than one receiving antenna. The coherent MIMO beamforming can rely on a synthesizer that generates chirps using the binary phase modulation method, which allows simultaneous transmission from each antenna for a better signal-to-noise ratio (SNR).

[0088] Obviously, the invention can be implemented with simple antennas, without MIMO technology, with other technologies.

[0089]

[0090] According to another aspect of the invention, a method is proposed for generating training data for an artificial intelligence-based emotion model, this method comprising the following steps for a subject:

[0091] - production of a training signal of duration T0 containing inducing stimuli of emotions that the subject receives,

[0092] - emission of a first radio frequency signal during the duration T0 in the direction of subject,

[0093] - reception of a second radio frequency signal containing a physiological signal due to the reflection of said first radio frequency signal on the subject,

[0094] - digital processing of the second radio frequency signal so as to generate respiratory and / or cardiac data.

[0095] The method according to the invention further comprises the following steps:

[0096] - definition of a time window Fl of duration Tl less than T0,

[0097] - application of several time windows Fl in a sliding manner on the respiratory and / or cardiac data; emotion model training data including data from each Fl time window,

[0098] - definition of a time window F2 of duration T2 less than T1, and of a window F3 of duration T3 less than T2,

[0099] - during the emission of the first radio frequency signal, detection of events in origin of the subject or calculated based on the subject's behavior;

[0100] - at each event, application of several F3 time windows in a way sliding over respiratory and / or cardiac data from a time window F2 preceding the event; emotion model training data including data from each time window F3.

[0101]

[0102] By events originating from the subject or calculated based on the subject's behavior, we mean any event initiated by the subject or not. This may be a response from the subject via an input interface of a processing unit or a signal automatically generated by a subject activity monitoring tool. This event may be calculated based on results from other physiological sensors, for example, GSR (Galvanic Skin Resistance) sensors, EEG (Electroencephalography) sensors, EMG (Electromyography) sensors, or PPG (Photoplethysmography) sensors.

[0103] According to an advantageous feature of the invention, the training signal can be a virtual reality environment in which the subject is immersed. This training signal can be any other video and / or audio signal, but preferably a 3D virtual reality environment.

[0104] A training signal is defined as a signal containing one or more stimuli designed to induce a predetermined emotion in the subject. This signal includes stimuli but also calm, emotionless moments, allowing for the generation of comparison labels with respect to the stimuli.

[0105]

[0106] According to one embodiment of the invention, the drive signal may to understand one or more stimuli capable of triggering a single emotion of stress in the subject.

[0107] With the invention, we thus provide for a real-time characterization of the dynamics on a single given dimension, for example stress, unlike for example a discrete classification of four emotions over a long time window of for example 2 minutes as in the prior art.

[0108] Other characterization techniques can be used such as continuous dimensions Valence and Intensity in indirect measurement, then translated into emotional categories or level categories (low, medium, high stress).

[0109]

[0110] According to an advantageous feature of the invention, the training signal can comprise several time-spaced stimuli, each stimulus being capable of triggering the same predetermined emotion; the stimuli being differentiated by their level of difficulty, complexity, repetitiveness, attentional effort, suddenness or duration.

[0111] The use of multimodal stimuli is planned. These stimuli are designed to evolve according to a controlled independent variable that varies along a unidimensional quantitative axis. This variable may be a level of difficulty, complexity, repetitiveness, attentional effort, suddenness, or task duration that can be objectively measured or estimated by evaluators and that has an impact on physiology without requiring verbal feedback from the subject. Such a stimulus can induce an associated internal state such as stress, cognitive load, concentration, tunnel vision, boredom, drowsiness, fatigue, surprise, or fear.

[0112] In other words, all stimuli can, for example, induce a certain type of stress by varying, in particular, the difficulty of the task experienced by the subject in order to induce periods of more or less intense stress, but it is also possible to consider inducing different types of emotions: stress, fear, joy, etc. In general, periods of calm are also provided to allow for classification.

[0113] Advantageously, the duration T2 can be between 1 second and 30 seconds; the duration T3 can be a minimum of 375 milliseconds, or between 375 milliseconds and 30 seconds, or between 1 and 5 seconds, with a minimum step of 1 millisecond.

[0114] The minimum duration of T3 corresponds to a single heartbeat of a newborn, for example.

[0115] For each response from the subject, several F3 time windows are applied in a sliding manner, with a fixed time offset or not.

[0116] A training signal lasting 3 minutes (T0) can be considered. The subject is free to respond to an emotional feeling whenever they wish, particularly in a way regularly, and especially when he experiences a strong emotional change.

[0117] The long window can have a duration of 1 min (Tl), and be applied in a sliding manner in 3 second steps throughout the entire observation period of the subject.

[0118] For the short window, we can pre-segment a period of interest of 10 seconds (T2) upstream of each response triggered by the subject, and within this period of interest, we retrieve the data from the short time windows of 3 seconds (T3) in steps of 500 ms for example, which makes 15 extracts of 3 seconds.

[0119] With the method according to the invention, after being trained, the emotion model can be deployed on an edge computing or embedded platform connected to radar sensors pointed at a targeted workspace. Once the human target is detected, the solution begins an initialization step, refining the model on this specific target with adjusted hyperparameters and architecture based on the initial data captured and additional data from optional supplementary sensors.

[0120] In general, Tl can be greater than 30 seconds and T3 can be less than 30 seconds.

[0121] According to yet another aspect of the invention, a system for detecting an emotion in a subject is proposed, comprising:

[0122] - a processing unit equipped with an emotion model and configured to put into implements a method for detecting an emotion in a subject as described above,

[0123] - at least one transmitting antenna connected to the processing unit and arranged to emit the first radio frequency signal in the direction of the subject,

[0124] - at least one receiving antenna connected to the processing unit and arranged to receive the second radio frequency signal containing the physiological signal due to the reflection of said first radio frequency signal on the subject.

[0125]

[0126] The system according to the invention includes the following advantages:

[0127] - non-intrusive: unlike camera-based systems, which require the Unlike millimeter-wave radar, which captures visual images of individuals, this type of radar operates in the radio frequency spectrum. This makes the system non-intrusive and respects privacy, as it does not capture any identifiable visual information such as facial features, skin color, or clothing.

[0128] - robustness to lighting conditions: camera-based systems are Sensitive to lighting conditions, changes in lighting can significantly affect the accuracy of emotion recognition. However, millimeter-wave radar operates independently of lighting conditions because it uses radio waves that are unaffected by changes in ambient light, making it suitable for both indoor and outdoor environments.

[0129] - insensitive to occlusions: the cameras can be obstructed by various objects, such as than hands, accessories, or other people, resulting in inaccurate emotion recognition. In contrast, millimeter-wave radar is capable of penetrating certain materials, including fabric, plastic, and thin barriers, making it less susceptible to occlusions and providing more reliable emotion recognition results.

[0130] - focused on internal physiology: although it may be easy for a subject to While controlling facial and behavioral expressions of emotions, the body's physiology can hardly be consciously controlled, and therefore the present invention can hardly be deceived.

[0131]

[0132] According to the invention, the system may include a relay box disposed near the transmitting and receiving antennas; the processing unit being remote and connected to the relay box via an internet-type communication network; the processing unit communicating with the transmitting and receiving antennas via the relay box.

[0133] Proximity means a distance allowing communication with or without wires to the antennas with a signal quality allowing the proper functioning of the system.

[0134] The processing unit can also be mounted locally and connected directly with the transmitting and receiving antennas.

[0135]

[0136] The fields of application of the invention may include:

[0137] - industry: the present invention can be installed on workstations in Combined with the company's system and a dedicated rules engine, this approach supports operators in their daily work and improves their well-being. The goal can be to foster interaction, thereby reducing workplace stress. This translates into less absenteeism, fewer accidents, increased productivity, and lower indirect costs.

[0138] - design: integrated into the development phase of a product or an enviable In this respect, the present invention allows at each iteration to consider the human factor in order to evaluate areas of friction and optimize the interaction between users and the product.

[0139] - training: integrated into the training tool, whether the training is face-to-face, in Whether virtual or remote, the present invention allows the trainer to better understand the internal emotional state of the participants in order to personalize the format and content in the transfer of knowledge.

[0140] - social robotics: social robots must be able to understand the emotion of their interlocutors to adapt their behavior and thus facilitate human-machine interactions while promoting their acceptance.

[0141] - healthcare: the present invention can assist medical and social personnel to They can deliver a difficult diagnosis by providing additional information about the patient's inner feelings. They can also pick up on the inner feelings of a patient or young child who is unable to verbalize them.

[0142] - Animal health: the present invention can help in stress management in animals domestic animals and farm animals destined for slaughter.

[0143] - entertainment: coupled with a rules engine, the present invention can take takes into account a player's emotions to offer a more personalized "gameplay" or video game scenario.

[0144] - the smart home: coupled with a decision rules engine, the present The invention can be installed in living and working spaces to modify the aesthetic and sensory environment according to the feelings of its occupants.

[0145]

[0146] Other features and advantages of the invention will become apparent from the detailed description of implementations and embodiments, which are by no means limiting, with reference to the accompanying figures in which:

[0147] [Fig-1] - [Fig. 1] is a schematic view illustrating the overall process for the generation according to the invention of cardiac characteristics as training data for an artificial intelligence model;

[0148] [Fig.2] - [Fig.2] is a schematic view illustrating the use according to the invention large and small time windows during the processing of radio frequency signals from the user;

[0149] [Fig.3] - [Fig.3] is a schematic view illustrating the overall process for the prediction of an emotion from an emotion model according to the invention;

[0150] [Fig.4] - [Fig.4] is a schematic view illustrating data generation training (in training phase) and useful data (in operation phase) from data representative only of cardiac activity according to the invention;

[0151] [Fig.5] - [Fig.5] is a schematic view illustrating data generation training (in training phase) and useful data (in operation phase) from data representative only of respiratory activity according to the invention;

[0152] [Fig.6] - [Fig.6] is a schematic view illustrating data generation training (in training phase) and useful data (in operation phase) from representative data of cardiac activity and representative data of respiratory activity according to the invention;

[0153] [Fig.7] - [Fig.7] is a flowchart illustrating the steps of a treatment digital radar signal processing for obtaining data representative of respiratory activity and data representative of cardiac activity according to the invention; and

[0154] [Fig.8] - [Fig.8] is a general view illustrating the working environment of a user lizer, environment in which components are arranged for the implementation of the process according to the invention.

[0155]

[0156] The embodiments described below are not in any way limiting; in particular, variants of the invention may be implemented comprising only a selection of the features described below, isolated from the other features described, if this selection of features is sufficient to confer a technical advantage or to differentiate the invention from the prior art. This selection includes at least one preferably functional feature without structural details, or with only a portion of the structural details if this portion alone is sufficient to confer a technical advantage or to differentiate the invention from the prior art.

[0157] Generally speaking, the design of an artificial intelligence model includes a training phase during which the model is fed a large amount of specific training data. The goal is to explore as many situations as possible so that the machine develops its predictive capacity.

[0158] Figure 1 relates to the training phase. A user 1 is shown equipped with a virtual reality headset 2. This headset 2 is connected to a processing unit not shown in Figure 1.

[0159] A training signal 3 is played back in the headset 2. Preferably, the training signal 3 is a 3D environment comprising videos and audio in which the user 1 is immersed, and depending on the situation, with which he can possibly interact.

[0160] Stimuli are integrated into this environment and are designed to create a predetermined emotion in the user. These stimuli may be different and placed at different times during the playback of the training signal. For example, the training signal playback duration T0 may be set at 3 minutes. Other durations may be considered, close to one minute or several minutes, up to, for example, 120 minutes.

[0161] One can consider a stimulus capable of provoking the same emotion and to which variations are applied throughout the reading. This is a multimodal stimulus which varies, for example, according to complexity.

[0162] When reading the training signal, the user may be asked to The system signals as soon as it experiences an emotion. This signaling can be done by any means, including manipulating an input on the processing unit. This input can be a mouse, a keyboard, or any other interface.

[0163] During the playback of the training signal, a first radio frequency signal 4 is sent towards user 1. The reflection of this first radio frequency signal, particularly off user 1, produces a second radio frequency signal 5 which is received for processing. In [Fig. 1], a transmitting antenna Tx emits the first radio frequency signal 4 and a receiving antenna Rx receives the second radio frequency signal 5.

[0164] Preferably, the transmitting antenna Tx and the receiving antenna Rx are part of a frequency-modulated continuous wave (FMCW) radar operating in the 57 GHz to 64 GHz or 76 GHz to 81 GHz range with a maximum bandwidth of 4 GHz. The FMCW radar unit is based on the coherent MIMO (Multiple Input Multiple Output) system with multiple Tx and Rx antennas. Other technologies may be used, such as ultra-wideband. Coherent MIMO beamforming relies on a synthesizer that generates chirps (n-frequency and amplitude-modulated pseudo-periodic signals) using binary phase modulation, which allows simultaneous transmission from each antenna for a better signal-to-noise ratio (SNR). The transmitted chirps are dynamically programmed, and the frames are divided into several subframes.Two simultaneous subframes are configured independently via a serial peripheral interface (SPI) by controlling the burst period, subframe period, loop number, and duty cycle. This configuration flexibility allows for the detection of micromovements within the radar coverage area. The configuration and generation of these subframe profiles are performed by a chirp module and a buffer within the processing unit.

[0165] The second received radio frequency signal contains physiological data of user 1. The processing 6 on this radio frequency signal 5 aims to determine a signal 6a representative of respiratory activity and a signal 6b representative of cardiac activity. A window F1 is applied to the signal 6a representing respiratory activity so as to generate data DF1 useful as training data for the artificial intelligence model 8. A window F3 is applied to the signal 6b representing cardiac activity so as to generate data DF3 useful as training data for the artificial intelligence model 8. The data DF1 and DF3 contain cardiac and respiratory characteristics of user 1 that the training of the artificial intelligence model 8 will highlight.

[0166] Many measurements are carried out with many users for a single or different predetermined known emotions. In [Fig. 2], a timeline from 0 to TO is shown, corresponding to the duration of training signal 3 in [Fig. 1]. This training signal can be a 3-minute video signal. Markers 9 and 10 represent moments when the user, immersed in the virtual environment, indicates having experienced an emotion. The user is encouraged to express themselves whenever they wish, especially when experiencing a strong emotional change.

[0167] According to the invention, a large window Fl, for example of a duration of 1 minute, is applied in a sliding manner, in 3-second increments, starting at the beginning of the video, i.e., 41 extracts per video. In [Fig. 1], the signal DF1 represents the data corresponding to the windows FL

[0168] A short window F2 is also applied, which is a 10-second period of interest preceding each of the user's responses 9 and 10. Within this period of interest, short windows F3, for example, of equal duration of 3 seconds, are applied in 500 ms increments, starting at the beginning of window F2, i.e., 15 extracts per window F2, on which training will detect cardiac characteristics.

[0169]

[0170] Once the artificial intelligence model has been trained and parameterized, it constitutes the emotion model that can be used in operation.

[0171] In [Fig. 3], an active user 11 is shown continuously receiving the first radio frequency signal 4. The second radio frequency signal 5 from the user is captured and processed 6 in the same way as during training. The difference between the training phase and the operational phase lies primarily in the use of time windows.

[0172] The time windows Fl are applied in a sliding and continuous manner throughout an entire user observation period. Preferably, they are no longer limited to the duration T0 of the training signal.

[0173] The F2 time windows are not used during the operational phase. Instead, the F3 time windows are applied in a sliding and continuous manner.

[0174] Thus, in operation, the time windows Fl and the time windows F3 are applied continuously throughout a user observation period.

[0175] The observation period is not necessarily continuous throughout the day. This observation period can be triggered based on information captured from the environment. For example, activating the processing and / or application of time windows only during difficult stages of a task being performed by the user, or only when an external alert occurs, for example, an emergency situation.

[0177] The DF1 and DF3 data on [Fig.3] feed an emotion model 12 which is therefore able to identify cardiac and respiratory characteristics to allow a real-time prediction of an emotion 13. The model 12 can advantageously be derived from the training of the training model 8 of [Fig. 1].

[0178] Fig. 4 is a schematic view illustrating the generation of training data (during training phase) and useful data (during operation phase) from data representative only of cardiac activity according to the invention.

[0179] Indeed, the invention can only be implemented using data representative of cardiac activity.

[0180] At the end of the digital processing 6, the respiratory activity 6a can be determined and not used or it can simply not be determined at all.

[0181] Cardiac activity 6b is split into two branches to allow for parallel processing. The F3 window is applied to the first branch, with the limitations of the F2 windows during the training phase and without the limitations of the F2 windows during the exploitation phase. The DF3 data from the application of each F3 window feeds into the training model or the emotion model.

[0182] On the second branch, the Fl window is applied, with a duration limit of T0 if we are in the training phase and no duration limit of T0 when we are in the exploitation phase. The DF la data directly resulting from the application of the Fl window can directly feed the training model 8 or the emotion model 12.

[0183] The invention also provides an embodiment where the DF la data directly from the application of the Fl window are first transformed in step 14 into one or more DFlb values ​​before feeding the training model 8 or the emotion model 12.

[0184] The invention further provides an embodiment where the DF la data directly from the application of the Fl window and the DFlb data from the transformation of the DFla data in step 14 together feed the training model 8 (in training phase) or the emotion model 12 (in exploitation phase).

[0185] In step 14, the transformation carried out is the calculation of a value of the heart rate variability HRV.

[0186] Figure 5 is a schematic view illustrating the generation of training data (during the training phase) and useful data (during the operational phase) from data representing only cardiac activity according to the invention. The same operations are performed as in Figure 4, except that here the initial data 6a relates only to respiratory activity. Another difference is the transformation step 14, during which at least one of the following values:

[0187] - a respiratory rate variability (RRV for “Respiratory Rate Variability” in English),

[0188] - a respiratory pattern (BPA for “Breathing Pattern Analysis” in English),

[0189] - a respiratory amplitude and its variability (RAV for “Respiratory Amplitude and Variability” in English).

[0190] Fig. 6 is a schematic view illustrating data generation training (in training phase) and useful data (in operation phase) from representative data of cardiac activity and representative data of respiratory activity according to the invention.

[0191] The same operations are performed as in [Fig. 4], except here the starting data are respiratory activity data 6a and cardiac activity data 6b. On one branch, the Fl window is applied to all data 6a and 6b, and on the other branch, the F3 window is applied to all data 6a and 6b.

[0192] Thus at step 14 of transformation, all the values ​​described with reference to figures 4 and 5 can be calculated, but a value of respiratory sinus arrhythmia (RSA for “Respiratory Sinus Arrythmia” in English can also be calculated).

[0193]

[0194] Figure 7 shows an example of a flowchart illustrating the steps of a digital processing of radar signals from the user to obtain data representative of respiratory activity and data representative of cardiac activity according to the invention. This is processing step 6 of Figure 1.

[0195] The treatment starts at step 15.

[0196] In step 16, the FMCW signals from the receiving antennas are received before conversion to an intermediate frequency in step 17.

[0197] An analog-to-digital conversion is performed in step 18. Digital signal processing is carried out using one or two real-time fixed- or floating-point digital signal processors that handle several instructions in parallel: interference reduction using DC component suppression (leakage between Tx and Rx), noise threshold increase detection, quantization noise reduction due to analog-to-digital conversion, constant false alarm rate (CFAR) detection, windowing for an FFT and a logarithmic scale transform of the signals.

[0198] In step 19, infinite impulse response filtering reduces electronic noise and extracts the specific components of the signal. Quadrature conversion is based on decomposition into two orthogonal components: a phase component (I) and a quadrature component (Q). This allows for the analysis of both the amplitude and phase of the signal.

[0199] In step 20, a data cube construction of the FMCW signals is carried out through three-dimensional matrices defined by the characteristics of the MIMO radar (number of channels linked to the number of antennas) and of the FMCW modulation (time and chirp).

[0200] In step 21, a Range-Doppler-Angle matrix is ​​developed from the data cubes. A succession of 1, 2 and 3 dimension Fourier transforms is applied for the estimation of the 3 pieces of information: distance, speed and angle of the target relative to the radar.

[0201] In step 22, a clutter suppression algorithm is implemented. Through a (time-sliding) transmission and reception window applied to the desired signatures, static states and slow movements, which are different from the desired modulations—such as the rib cage, cardiac and respiratory activity—are excluded. The interference between the harmonics and the fundamental frequency of the heart and respiration is improved by implementing, in step 23, an empirical decomposition algorithm into ensemble modes and, in step 24, decomposing the signal into an Intrinsic Mode Function (IMF).

[0202] In step 25, a continuous wavelet transform (CWT) is performed to improve the identification of cardiac and respiratory rhythms. This is an adaptive filtering that generates 6a data representative of respiratory activity and 6b data representative of cardiac activity.

[0203] Data 6a representing respiratory activity undergo normalization in step 26. Data 6b representing cardiac activity also undergo normalization in step 27. Normalizations 26 and 27 are optional and may be identical or different.

[0204] In [Fig. 8], user 11 is shown seated at a workstation 28 in a workshop. Transmitting and receiving antennas can be positioned at various locations 29 around user 11 in the workshop. A processing unit 30 can be provided for implementing the method according to the invention. This processing unit can also be directly integrated into a module with the antennas. This processing unit includes the software and hardware suitable, in particular, for managing the antennas and the signals passing through them. The processing unit is connected to the antennas either wirelessly or via a cable. Alternatively, the processing unit can be positioned remotely, and a relay box can be placed in the workshop to relay the signals between the processing unit and the antennas. Such a processing unit can be powerful enough to manage several sets of antennas for multiple users.

[0206] The present invention therefore relates to a method of emotion recognition combining radio frequency radar models and artificial intelligence to derive emotion from the non-contact measurement of at least one key physiological signal of at least one person.

[0207] The invention makes it possible to extract the complete cardiac and respiratory signals according to the short and long time windows chosen, but also classic characteristics of cardiac and respiratory activities such as HRV, RRV, BPA, RAV, RSA and other characteristics within the same heartbeat or in relation to the relationship between different cardiac characteristics of successive heartbeats.

[0208]

[0209] The various embodiments of the present invention comprise several steps. These steps can be implemented by instructions from an executable machine using, for example, a microprocessor.

[0210] Alternatively, these steps can be carried out by specific integrated circuits comprising hardwired logic to execute the steps, or by any combination of programmable and custom components.

[0211] The present invention can also be supplied in the form of a computer program product which may include a non-transient computer memory medium containing instructions executable on a computer machine, these instructions being able to be used to program a computer (or any other electronic device) to carry out the process.

[0212]

[0213] Of course, the invention is not limited to the examples just described and many modifications can be made to these examples without departing from the scope of the invention.

[0214]

Claims

Claims

1. Method for detecting an emotion in a subject (11) comprising the following steps: - emitting a first radio frequency signal (4) towards the subject, - receiving a second radio frequency signal (5) containing a physiological signal due to the reflection of said first radio frequency signal on the subject, - digital processing (6) of the second radio frequency signal (5) so as to generate respiratory data (6a) and / or cardiac data (6b), - applying several time windows Fl of duration Tl in a sliding manner to the respiratory data and / or the cardiac data, - applying several time windows F3 of duration T3 in a sliding manner to the respiratory data and / or the cardiac data; the duration T3 being less than the duration Tl, and - predicting an emotion by an emotion model (12) based on artificial intelligence from the data coming from the windows Fl and the data coming from the windows F3.

2. Method according to claim 1, characterized in that the data coming from the time window F1 are directly transmitted to the emotion model (12) and / or undergo a calculation (14) of a value of at least one respiratory and / or cardiac biomarker, this value being transmitted to the emotion model (12).

3. Method according to claim 2, characterized in that the calculation (14) of a value of at least one respiratory and / or cardiac biomarker comprises at least one of the following calculations: - calculation of a heart rate variability (HRV for “heart rate variability”), - calculation of a respiratory variability (RRV for “Respiratory Rate Variability”)) or of a respiratory pattern (BPA for “Breathing Pattern Analysis”) or of a respiratory amplitude and its variability (RAV for “Respiratory Amplitude and Variability”), - calculation of a respiratory sinus arrhythmia (RSA for “Respiratory Sinus Arrythmia” in English).

4. Method according to any one of the preceding claims, characterized in that the duration T1 is at least 30 seconds with a step sliding time of at least 1 millisecond; or a duration Tl of between 30 seconds and 5 minutes with a sliding step of between 1 millisecond and 15 seconds; or a duration Tl of between 30 seconds and 1 minute 15 seconds with a sliding step of between 3 seconds and 5 seconds.

5. Method according to any one of the preceding claims, characterized in that the duration T3 is at least 375 milliseconds, or between 375 milliseconds and 30 seconds, or between 1 and 5 seconds; with a step of at least 1 millisecond.

6. Method according to any one of the preceding claims, characterized in that the first radio frequency signal (4) is a frequency-modulated continuous wave radar signal (FMCW) or an ultra-wideband signal (UWB).

7. Method according to any one of the preceding claims, characterized in that the respiratory data and / or the cardiac data undergo a normalization step (26, 27) before application of the time windows.

8. Method according to any one of the preceding claims, characterized in that the emotion model (12) consists of one or more simple or deep learning models.

9. Method according to any one of the preceding claims, characterized in that the emotion model (12) comprises a hybrid recurrent neural network.

10. Method according to claim 9, characterized in that the emotion model (12) comprises a fusion between a closed recurrent unit GRU (for “Gated recurrent unit” in English) and a long-term and short-term memory network LSTM (for “Long Short-term Memory” in English).

11. Method according to any one of the preceding claims, characterized in that the emotion model (12) comprises: - a slow dynamics model dedicated to the data coming from the time windows F1, - a fast dynamics model dedicated to the data from the time windows F3, and - a decision-making algorithm common to the slow dynamics model and to the fast dynamics model.

12. A method according to any preceding claim, ca- characterized in that the transmission of the first radio frequency signal (4) is carried out by means of several transmission antennas (29) and the reception of the second radio frequency signal is carried out by means of several reception antennas (29) according to MIMO technology (for "Multiple-Input Multiple-Output" in English).

13. System for detecting an emotion in a subject (11) comprising: - a processing unit (30) provided with an emotion model (12) and configured to implement a method according to any one of the preceding claims, - at least one transmitting antenna (29) connected to the processing unit and arranged to transmit the first radio frequency signal towards the subject (11), - at least one receiving antenna (29) connected to the processing unit and arranged to receive the second radio frequency signal containing the physiological signal due to the reflection of said first radio frequency signal on the subject.

14. System according to claim 13, characterized in that the processing unit (30) is locally embedded and connected directly with the transmitting and receiving antennas (29).

15. System according to claim 13, characterized in that it comprises a relay box arranged close to the transmission and reception antennas (29); the processing unit (30) being remote and connected to the relay box via an internet-type communication network; the processing unit (30) communicating with the transmission and reception antennas via the relay box.

Images