Method for generating training data for an artificial intelligence-based emotion model

Abstract

The present invention relates to a method for generating training data for an artificial intelligence-based emotion model. This method comprises processing a radio frequency signal containing physiological data to obtain cardiac characteristics for use as training data for the emotion model. The processing includes applying large and short time windows to the radio frequency signal to determine the cardiac characteristics. See Figure 1.

Description

Title of the invention: Method for generating training data for an emotion model based on artificial intelligence

[0001] The present invention relates to a method for generating training data for an emotion model based on artificial intelligence. The invention also relates to a method for exploiting the emotion model once trained as well as a system comprising a processing unit for implementing such an emotion model.

[0002]

[0003] The invention finds a particularly interesting application in the detection of emotion of a user on a workstation. But the invention has a broader scope since it can be applied to many other professional or particular fields such as human-machine interaction, virtual reality, games and health care.

[0004]

[0005] Society continues to evolve towards an increasingly present integration of technology in our daily lives.

[0006] Our relationship with consumption, health, education, information, and even the very functioning of our society today requires ever-greater mastery of the digital world. It must be said that the technological progress of recent decades and the advent of the Internet and a more globalized economy have greatly facilitated access to and use of highly technological products, from a financial and functional point of view.

[0007] And yet, digital interfaces remain generally passive with regard to their users and what drives them: their intentions, their emotions beyond their interactions. They are unable to recognize episodes of frustration when the user is confronted with a poorly designed web interface. It is impossible to monitor the concentration and fatigue of an operator on a high-risk industrial workstation. It is impossible to recognize the despair of a student overwhelmed with knowledge in his learning or, on the contrary, to celebrate a moment of happiness with a player proud of his prowess in a video game. If these interfaces were capable of sensing the emotions and intentions of their users, we would no longer be obliged to use explicit commands and would benefit from a more natural and therefore even more accessible interaction.In addition, machines would demonstrate their ability to support humans in regulating their own emotions, which is... would result in greater security and well-being.

[0008]

[0009] Today, most approaches in affective computing rely on the exploitation of audiovisual information such as facial expressions, posture or prosody. They are very useful because they are easily accessible, but they are not free from potential biases. Since facial expressions are a mode of social communication, they are rarely expressed by a user alone in front of their screen. Conversely, they may be forced to hide an emotion of opposite valence. The same applies to the expression of emotions through voice. Moreover, facial expressions are not necessarily visible in the camera field and their detection may be limited in an uncontrolled environment due to lighting conditions.

[0010]

[0011] In order to overcome these limitations, we can look at more physiological and therefore more unconscious signals induced by the regulation of the autonomic nervous system closely linked to emotional changes. Systems are known that use an electrocardiogram on the chest to monitor cardiac activity. Electrodes distributed over the body can be used to measure galvanic conductance and therefore variations in perspiration. An electroencephalogram can be used to characterize neuronal activity, while an oximeter can be used to monitor a user's pulse. By processing these signals with trained artificial intelligence modules, it is possible to extract a certain dynamic of the subject's state and feelings. Such measurement solutions, practical in the laboratory, quickly prove uncomfortable in the everyday environment.

[0012] Non-invasive sensors are known for use in emotion recognition. The use of thermal cameras to capture surface temperature differences on the face is becoming increasingly popular. While this approach has the merit of being focused on the physiological component of emotion, it nevertheless has the constraint of requiring the face of the target to be visible.

[0013]

[0014] In recent years, some researchers have also used radar technologies on certain radio frequency ranges to remotely detect micro-movements of the rib cage and indirectly of the heart muscles. From the processing of these micro-body movements, they have accessed real-time changes in heart rate which contain diagnostic information for the evaluation of emotions.

[0015] Document US20170311901A1 is known describing a radar system for emotion recognition. The radar used is a modulated continuous wave radar in FMCW frequency (for “Frequency-Modulated Continuous Wave” in English). The characterization of emotions is done by a discrete classification of four emotions over a 2-minute time window.

[0016] We know the document Siddiqui HUR et al., (2021), “Respiration Based Non-Invasive Approach for Emotion Recognition Using Impulse Radio Ultra Wide Band Radar and Machine Learning.”, Sensors, 21(24), Article 24. https: / / doi.org / 10.3390 / s21248336, describing a method for extracting the number of breaths per minute from a signal coming from a UWB radar.

[0017] We know the document Khan et al., (2021), “Deep learning framework for subject-independent emotion detection using wireless signals.”, PLOS ONE, 16(2), e0242946. https: / / doi.org / 10.1371 / journal.pone.0242946, describing a radar system for emotion recognition. Emotion characterization is done by discrete classification of several emotions over long time windows (2 min). It describes an artificial intelligence model built directly from the raw data of the radar signal.

[0018]

[0019] We know the document Dang et al., (2022), “Emotion recognition method using millimeter wave radar based on deep learning.”, IET Radar, Sonar & Navigation, 16(11), 1796-1808. https: / / doi.org / 10.1049 / rsn2.12297, describing a 77 GHz FMCW millimeter radar for emotion recognition. This document describes signal processing to extract cardiac and respiratory activities, both taken into account in the model and combined for classification.

[0020]

[0021] The present invention aims at a new real-time method of recognizing emotions by using radar.

[0022] Another aim of the invention is precise and reliable recognition of emotions.

[0023] At least one of the objectives is achieved with a method for generating training data for an artificial intelligence-based emotion model, this method comprising the following steps for a subject:

[0024] - production of a training signal of duration T0 containing inducing stimuli of emotions that the subject receives,

[0025] - emission of a first radio frequency signal during the duration T0 in the direction of the subject,

[0026] - reception of a second radio frequency signal containing a physiological signal due to the reflection of said first radio frequency signal on the subject,

[0027] - digital processing of the second radio frequency signal so as to generate respiratory data and / or cardiac data.

[0028] The method according to the invention further comprises the following steps:

[0029] - definition of a time window Fl of duration Tl less than T0,

[0030] - application of several time windows Fl in a sliding manner on the respiratory data and / or cardiac data; emotion model training data including data from each time window Fl,

[0031] - definition of a time window F2 of duration T2 less than T1, and of a window F3 of duration T3 less than T2,

[0032] - during the transmission of the first radio frequency signal, detection of events in origin of the subject or calculated on the basis of the subject's behavior;

[0033] - to each event, application of several time windows F3 in such a way sliding on respiratory data and / or cardiac data from a time window F2 preceding the event; the emotion model training data comprising data from each time window F3.

[0034]

[0035] The method according to the invention offers a non-invasive solution, usable in a natural environment and capable of recognizing the emotional dynamics of a subject as close as possible to their physiology without requiring the wearing of accessories in contact with the body.

[0036] With the use of large windows and small windows, the characterization of cardiac data and the extraction of components of interest to feed the associated AI model are improved.

[0037] The analysis of the large windows makes it possible to consider the overall physiological activity throughout the reading of the training signal by the subject.

[0038] The analysis of small windows makes it possible to consider more local activities specific to an emotional event and to characterize in the signal resulting from physiological activity specific patterns of more local emotional events to be detected.

[0039] The time windows F1 preferably have the same fixed duration, but it is possible to envisage time windows F1 with different durations. The same applies to the time windows F2 or F3.

[0040] Carrying out analysis of physiological signals over short time windows preceding a response from the subject makes it possible to obtain predictions over short periods, which subsequently allows real-time prediction of sudden emotional events during the exploitation phase of the emotion model.

[0041] We can thus evaluate the dynamics of emotions by targeting the intensity of one or more specific emotions or internal states or cognitive intentions over time on a human or animal subject.

[0042] Physiological data includes, for example, but is not limited to, heart rate fluctuations, respiratory fluctuations, eye activity, eye responses, and other data. galvanic skin and blood pulses.

[0043]

[0044] With the method according to the invention, two dynamics of emotional physiology are considered: a slow dynamic corresponding to progressive variations of the targeted emotional dimension, such as stress or joy, and a much faster dynamic, in the space of a few seconds or less, corresponding to more sudden and local variations of this same emotional dimension.

[0045] It is considered that the information specific to the emotion experienced occurs in physiological activity over latencies much shorter than 1 min, in the space of a few seconds. It is this information that is sought by using short time windows in addition to long windows for labeled data, the latter allowing, for example, the extraction of a reference emotional level.

[0046] By events originating from the subject or calculated on the basis of the subject's behavior, we mean any event initiated by the subject or not. This may be a response from the subject via an input interface of a processing unit or a signal generated automatically by a tool for monitoring the subject's activity. This event may be calculated on the basis of results from other physiological sensors, for example GSR (for “Galvanic Skin Resistance” in English) sensors, EEG (Electroencephalography) sensors, EMG (Electromyography) sensors, or PPG (Photoplethysmography) sensors.

[0047] According to an advantageous characteristic of the invention, the training signal may be a virtual reality environment in which the subject is immersed. This training signal may be any other video and / or audio signal but preferably a 3D virtual reality environment.

[0048] A training signal is understood to mean a signal containing one or more stimuli intended to induce a predetermined emotion in the subject. This signal includes stimuli but also calm moments, without emotions, allowing the generation of comparison labels with respect to the stimuli.

[0049]

[0050] According to one embodiment of the invention, the training signal may comprise one or more stimuli capable of triggering a single stress emotion in the subject.

[0051] With the invention, a characterization of the dynamics in real time is thus provided on a single given dimension, for example stress, unlike for example a discrete classification of four emotions over a long time window of for example 2 min as in the prior art.

[0052] Other characterization techniques can be used, such as continuous dimensions Valence and Intensity in indirect measurement, then translated into ca emotional categories or level categories (low, medium, high stress).

[0053]

[0054] According to an advantageous characteristic of the invention, the training signal can comprise several stimuli spaced apart in time, each stimulus being capable of triggering the same predetermined emotion; the stimuli being differentiated by their level of difficulty, complexity, repetitiveness, effort of attention, suddenness or duration.

[0055] The use of multimodal stimuli is envisaged. These stimuli are made to evolve according to a controlled independent variable which varies along a unidimensional quantitative axis. This may be a level of difficulty, complexity, repetitiveness, attention effort, suddenness or even a task duration which can be objectively measured or estimated by evaluators and which has an impact on physiology without requiring verbal feedback from the subject. Such a stimulus may lead to the induction of an associated internal state such as stress, cognitive load, concentration, tunnel effect, boredom, drowsiness, fatigue, surprise, fear.

[0056] In other words, all stimuli can, for example, induce a certain type of stress by varying in particular the difficulty of the task undergone by the subject to induce periods of more or less strong stress, but we can also envisage inducing different types of emotions: stress, fear, joy, etc. Generally speaking, we also provide periods of calm to enable classification.

[0057]

[0058] According to an advantageous characteristic of the invention, the data originating from the time window F1 can be directly used as training data and / or undergo a calculation of a value of at least one respiratory and / or cardiac biomarker, this value being used as training data for the emotion model.

[0059] In particular, the calculation of a value of at least one respiratory and / or cardiac biomarker may comprise at least one of the following calculations:

[0060] - calculation of heart rate variability (HRV for “heart rate va reliability”),

[0061] - calculation of respiratory variability (RRV for “Respiratory Rate Variability”) or of respiratory pattern analysis (BPA) or respiratory amplitude and variability (RAV),

[0062] - calculation of respiratory sinus arrhythmia (RSA) in English).

[0063] RSA is representative of the interaction between cardiac activity and respiratory activity, reflecting the activity of the parasympathetic autonomic nervous system.

[0064] Thus, the method according to the invention can be applied to cardiac activity alone, to respiratory activity alone or both activities.

[0065] Depending on the activity used, one or more of the functions listed above can be calculated.

[0066]

[0067] According to an advantageous characteristic of the invention, the duration T1 is at least 30 seconds with a sliding step of at least 1 millisecond; preferably the duration T1 is between 30 seconds and 5 minutes with a sliding step of between 1 millisecond and 15 seconds; ideally the duration T1 is between 30 seconds and 1 minute 15 seconds with a sliding step of between 3 seconds and 5 seconds.

[0068]

[0069] The duration Tl corresponds to the large time windows which are applied in a sliding manner, with a fixed time shift or not.

[0070] The physiological data obtained from these long windows, in particular beyond the minute, allow for example the calculation of heart rate variability (HRV) which requires waiting for several heartbeats to be calculated.

[0071]

[0072] Advantageously, the duration T2 can be between 1 second and 30 seconds; T3 being a minimum of 375 milliseconds, or between 375 milliseconds and 30 seconds, or between 1 and 5 seconds, with a step of minimum 1 millisecond.

[0073] For each response from the subject, several F3 time windows are applied in a sliding manner, with a fixed or unfixed time shift.

[0074] We can consider a training signal that lasts 3 min (T0). The subject is free to respond to an emotional feeling whenever he wishes, particularly on a regular basis and especially when he feels a strong emotional change.

[0075] The long window can have a duration of 1 min (Tl), and be applied in a sliding manner in steps of 3 seconds, starting at the beginning of the training signal, which makes 41 extracts per training signal, on which we analyze for example the variability of the heart rate (HRV).

[0076] For the short window, we can pre-segment a period of interest of 10 seconds (T2) upstream of each response triggered by the subject, and within this period of interest, we recover the data from the short time windows of 3 seconds (T3) in steps of 500 ms for example, which makes 15 extracts of 3 seconds.

[0077]

[0078] According to the invention, the first radio frequency signal may be a continuous wave radar signal modulated according to an FMCW protocol (for “Frequency-Modulated Continuous Wave” in English) or even an ultra wide band ULB signal (or UWB for “Ultra Wide Band” in English)

[0079] For example, the FMCW radar can operate in both frequency bands [57-64GHz] and [76-81GHz] up to 4 GHz. The UWB radar can operate in a range between 6 and 8.5 GHz.

[0080]

[0081] According to the invention, the respiratory data and / or the cardiac data can undergo a normalization step before applying the time windows.

[0082]

[0083] According to the invention, the data from cardiac and / or respiratory activities are processed independently through two long sliding windows F1 and short sliding windows F3 to extract one or more data from among: - All cardiac signals present in successive windows Fl - All respiratory signals present in successive windows Fl - Key cardiac measurements from FL temporalities: for example HRV (“Heart Rate Variability”) - Key respiratory measurements from FL temporalities: for example RRV (“Respiratory Rate Variability”), BPA (“Breathing Pattern Analysis”, RAV (“Respiratory Amplitude Variability”) - Measures reflecting the synchronization between cardiac and respiratory activity from FL temporalities, for example RSA (“Respiratory Sinus Arrythmia”) - All cardiac signals present in successive F3 windows - All respiratory signals present in successive F3 windows.

[0084] This data is then used to feed the training of AI models.

[0085] Additional data from other types of sensors than the aforementioned antennas can be used in operation to help weight the output results of the chosen Deep-Learning AI model.

[0086] With the invention, through the analysis of the characteristics of the time and frequency domain data, the periodic heart rate and respiratory data are extracted.

[0087] The outputs of radar sensors are representations of physiological activities in their own dimensional, temporal or frequency spaces embedded in the noise signal. The signal is filtered and processed to reveal as accurately as technically possible the physiological signal from the converted baseband signal.

[0088] A clutter removal algorithm can be implemented. Through a window (sliding in time) of transmission and reception applied to the signature sought, static states as well as slow movements, therefore different from the desired modulations: rib cage, cardiac and respiratory activity, are excluded. The interference between harmonics and the fundamental frequency of the heart and respiration can be improved by implementing an empirical mode set decomposition algorithm and decomposing the signal into an Intrinsic Mode Function (IMF). The improvement of the heart and respiratory rate can be ensured by the continuous wavelet transform (CWT).

[0089] The invention makes it possible to extract the complete cardiac and respiratory signals according to the chosen short and long time windows, but also classic characteristics of cardiac and respiratory activities such as HRV, RRV, BPA, RAV, RSA and other characteristics within the same heartbeat or in relation to the relationship between different cardiac characteristics of successive heartbeats.

[0090] The data is then optionally normalized and missing values ​​can be filled in if necessary. All of this data can then be optionally converted into vectors to be used as inputs in artificial intelligence-based emotion models.

[0091] According to an advantageous characteristic of the invention, the emotion model is made up of one or more learning models, simple, deep (“deep learning” in English) or preferably hybrid.

[0092] The hybrid recurrent neural network allows the artificial intelligence model to be trained and evaluated to detect and classify significant changes in cardiac and respiratory activities from the calculated cardiac and respiratory characteristics.

[0093] Preferably, the hybrid recurrent neural network results from a fusion between a closed recurrent unit GRU (for “Gated recurrent unit” in English) and a long-term and short-term memory network LSTM (for “Long Short-term Memory” in English).

[0094] Such a fusion is well suited for predictive and classification analysis of time series on a reasonably small data set. This fusion results in an improved and more accurate model. The GRU and LSTM networks are combined to take advantage of their respective strengths, including the ability of GRU to identify short-term dependencies and the ability of LSTM to identify long-term dependencies. By combining the LSTM and GRU models, the invention identifies different types of temporal patterns or dependencies present in the data, thereby improving the ability of the emotion model to make accurate predictions with preserved robustness and generalization capability. This also reduces the risks of over- adjustment and makes the whole more resilient to noise or uncertainties in the data.

[0095] In addition, the simplicity of the GRU improves the explainability of algorithmic decisions.

[0096] Advantageously, the emotion model can include:

[0097] - a slow dynamics model dedicated to data from tem windows porelles Fl,

[0098] - a fast dynamics model dedicated to F3 time window data, and

[0099] - a decision-making algorithm common to the slow dynamics model and to the fast dynamics model.

[0100]

[0101] According to an advantageous characteristic of the invention, in the context of using an FMCW radar, the transmission of the first radio frequency signal can be carried out by means of several transmission antennas and the reception of the second radio frequency signal can be carried out by means of several reception antennas according to MIMO technology (for “Multiple-Input Multiple-Output” in English).

[0102] The FMCW radar can therefore be based on a coherent MIMO system with more than one transmitting antenna and more than one receiving antenna. The coherent MIMO beamforming can be based on a synthesizer that generates chirps using the binary phase modulation method, which allows simultaneous transmission from each antenna for a better signal-to-noise ratio (SNR).

[0103] Obviously, the invention can be implemented with simple antennas, without MIMO technology, with other technologies.

[0104]

[0105] According to another aspect of the invention, there is provided a method for detecting an emotion in a subject comprising the following steps:

[0106] - emission of a first radio frequency signal towards the subject,

[0107] - reception of a second radio frequency signal containing a physiological signal due to the reflection of said first radio frequency signal on the subject,

[0108] - digital processing of the second radio frequency signal so as to generate respiratory data and / or cardiac data,

[0109] - application of several time windows Fl of duration Tl in a sliding manner on respiratory data and / or cardiac data,

[0110] - application of several time windows F3 of duration T3 in a sliding manner, with or without overlap, on respiratory data and / or cardiac data; the duration T3 being less than the duration Tl,

[0111] - prediction of an emotion by the emotion model from the data coming from Fl windows and data from F3 windows.

[0112]

[0113] With the method according to the invention, after having been trained, the emotion model can be deployed on an edge computing or embedded platform connected to radar sensors pointed at a targeted workspace. Once the human target is detected, the solution starts an initialization step by fine-tuning the model on this specific target with adjusted hyperparameters and architecture based on the first captured data and additional data from optional additional sensors.

[0114] Generally, T1 may be greater than 30 seconds and T3 may be less than 30 seconds.

[0115] The advantages and technical characteristics mentioned above during training and which are compatible can be repeated here.

[0116] According to yet another aspect of the invention, there is provided a system for detecting an emotion in a subject comprising:

[0117] - a processing unit provided with an emotion model and configured to implement implements a method for detecting an emotion in a subject as described above,

[0118] - at least one transmitting antenna connected to the processing unit and arranged to emit the first radio frequency signal towards the subject,

[0119] - at least one receiving antenna connected to the processing unit and arranged to receive the second radio frequency signal containing the physiological signal due to the reflection of said first radio frequency signal on the subject.

[0120]

[0121] The system according to the invention includes the following advantages:

[0122] - non-intrusive: unlike camera-based systems, which require the Capturing visual images of individuals, millimeter wave radar operates in the radio frequency spectrum. This makes the system non-intrusive and privacy-respecting, as it does not capture any identifiable visual information such as facial features, skin color, or clothing.

[0123] - robustness to lighting conditions: camera-based systems are sensitive to lighting conditions, and changes in lighting can significantly affect the accuracy of emotion recognition. However, millimeter wave radar works independently of lighting conditions because it uses radio waves that are not affected by changes in ambient light, making it suitable for both indoor and outdoor environments.

[0124] - insensitive to occlusions: cameras can be obstructed by various objects, such as than hands, props, or other people, resulting in inaccurate emotion recognition results. In contrast, millimeter wave radar is able to penetrate certain materials, including fabric, plastic, and thin barriers, making it less susceptible to occlusions and providing more reliable emotion recognition results.

[0125] - focused on internal physiology: although it may be easy for a subject to control the facial and behavioral expression of emotions, the physiology of the body can hardly be consciously controlled and therefore the present invention can hardly be deceived.

[0126]

[0127] According to the invention, the system may comprise a relay box arranged close to the transmission and reception antennas; the processing unit being remote and connected to the relay box via an internet-type communication network; the processing unit communicating with the transmission and reception antennas via the relay box.

[0128] Proximity means a distance allowing wired or wireless communication with the antennas with a signal quality allowing the system to function properly.

[0129] The processing unit can also be embedded locally and connected directly with the transmitting and receiving antennas.

[0130]

[0131] The fields of application of the invention may include:

[0132] - industry: the present invention can be installed on workstations in Combination with the company's system and a dedicated rules engine to support operators in their daily work and improve their well-being. The goal can be to promote interactions, thus reducing stress at work. This results in less absenteeism, fewer accidents, increased productivity, and reduced indirect costs.

[0133] - design: integrated into the development phase of a product or environment ronmentally, the present invention allows each iteration to consider the human factor in order to evaluate areas of friction and optimize the interaction between users and the product.

[0134] - training: integrated into the training tool, whether the training is face-to-face, in virtual or remote, the present invention allows the trainer to better understand the internal emotional state of the participants to personalize the format and content in the knowledge transfer.

[0135] - social robotics: social robots must be able to understand the emotion of their interlocutors to adapt their behavior and thus facilitate interactions between man and machine while promoting their acceptance.

[0136] - healthcare: the present invention can help medical and social personnel to announce a difficult diagnosis by providing additional information about the patients' internal feelings. It can also capture the inner feelings of a patient or a young child who is unable to verbalize them.

[0137] - animal health: the present invention can help in stress management in domestic animals and farm animals destined for slaughter.

[0138] - entertainment: coupled with a rules engine, the present invention can take takes into account a player's emotions to offer more personalized "gameplay" or video game scenario.

[0139] - the smart home: coupled with a decision rules engine, the present This invention can be installed in living and working spaces to modify the aesthetic and sensory environment according to the feelings of its occupants.

[0140]

[0141] Other characteristics and advantages of the invention will appear on reading the detailed description of implementations and embodiments which are in no way limiting, with regard to the appended figures in which:

[0142] [Fig-1] - [Fig. 1] is a schematic view illustrating the overall flow for the generation according to the invention of cardiac characteristics as training data for an artificial intelligence model;

[0143] [Fig.2] - [Fig.2] is a schematic view illustrating the use according to the invention large and small time windows when processing radio frequency signals from the user;

[0144] [Fig.3] - [Fig.3] is a schematic view illustrating the overall flow for the prediction of an emotion from an emotion model according to the invention;

[0145] [Fig.4] - [Fig.4] is a schematic view illustrating the generation of data training (in training phase) and useful data (in operating phase) from data representative only of cardiac activity according to the invention;

[0146] [Fig.5] - [Fig.5] is a schematic view illustrating the generation of data training (in the training phase) and useful data (in the operating phase) from data representative only of the respiratory activity according to the invention;

[0147] [Fig.6] - [Fig.6] is a schematic view illustrating the generation of data training (in the training phase) and useful data (in the operating phase) from data representative of cardiac activity and data representative of respiratory activity according to the invention;

[0148] [Fig.7] - [Fig.7] is a flowchart illustrating steps in a treatment digitalization of radar signals for obtaining data representative of respiratory activity and data representative of cardiac activity according to the invention; and

[0149] [Fig.8] - [Fig.8] is a general view illustrating the working environment of a user lizer, environment in which components are arranged for implementing the method according to the invention.

[0150]

[0151] The embodiments which will be described below are in no way limiting; it will be possible in particular to implement variants of the invention comprising only a selection of characteristics described below isolated from the other characteristics described, if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection comprises at least one preferably functional characteristic without structural details, or with only a part of the structural details if this part alone is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.

[0152] Generally speaking, the design of an artificial intelligence model involves a training phase during which an artificial intelligence model is fed with numerous specific training data. The objective is to explore as many situations as possible so that the machine develops the capacity for prediction.

[0153] [Fig.l] concerns the training phase. We can see a user 1 equipped with a virtual reality headset 2. This headset 2 is connected to a processing unit not shown in [Fig.l].

[0154] A training signal 3 is played in the headset 2. Preferably, the training signal 3 is a 3D environment comprising videos and audio in which the user 1 is immersed, and depending on the situations, with which he can possibly interact.

[0155] Stimuli are integrated into this environment and are intended to create a predetermined emotion in the user. These may be different stimuli placed at different times during the reading of the training signal. For example, a duration T0 of reading of the training signal equal to 3 min may be provided. Other durations may be considered close to a minute or several minutes, up to 120 minutes for example.

[0156] We can consider a stimulus capable of provoking the same emotion and to which we apply variations throughout the reading. This is a multimodal stimulus which varies, for example, according to complexity.

[0157] When reading the training signal, the user may be asked to report as soon as he feels an emotion. This reporting may be done by any means, in particular by manipulating an input of the processing unit. This input may be a mouse, a keyboard or any other interfacing means.

[0158] During the reading of the training signal, a first radio frequency signal 4 is sent towards the user 1. The reflection of this first radio frequency signal, in particular on the user 1, produces a second radio frequency signal 5 which is captured for processing. In [Fig.l], a transmitting antenna Tx transmits the first radio frequency signal 4 and a receiving antenna Rx captures the second radio frequency signal 5.

[0159] Preferably, the transmitting antenna Tx and the receiving antenna Rx are part of a FMCW frequency modulated continuous wave radar operating in a range between 57GHz and 64GHz or 76GHz and 81GHz with a maximum bandwidth of 4 GHz. The FMCW radar unit is based on the coherent MIMO (Multiple Input Multiple Output) system with several Tx antennas and several Rx antennas. Other technologies can be used such as ultra wideband. The coherent MIMO beamforming is based on a synthesizer that generates chirps (frequency and amplitude modulated pseudo-periodic signal) following the binary phase modulation method, which allows the simultaneous transmission of each antenna for a better signal-to-noise ratio (SNR). The transmitted chirps are dynamically scheduled and the frames are divided into several subframes.Two simultaneous subframes are configured independently via a serial peripheral interface (SPI) by controlling the burst period, subframe period, loop number, and duty cycle. This configuration flexibility allows for the detection of micro-motions within the radar coverage area. The configuration and generation of these subframe profiles is performed by a chirp module and a buffer within the processing unit.

[0160] The second radio frequency signal received contains physiological data of the user 1. The processing 6 on this radio frequency signal 5 aims to determine a signal 6a representative of the respiratory activity and a signal 6b representative of the cardiac activity. A window F1 is applied to the signal 6a representative of the respiratory activity so as to generate data DF1 useful as training data for the artificial intelligence model 8. A window F3 is applied to the signal 6b representative of the cardiac activity so as to generate data DF3 useful as training data for the artificial intelligence model 8. The data DF1 and DF3 contain cardiac and respiratory characteristics of the user 1 that the training of the artificial intelligence model 8 will make it possible to highlight.

[0161] Many measurements are carried out with many users for a single or different predetermined known emotions. In [Fig.2] we see a time line from 0 to T0 corresponding to the duration of the training signal 3 of [Fig.l]. This training signal can be a video signal which lasts 3 min. Markers 9 and 10 are times at which the user, immersed in the virtual environment, reports having felt an emotion. The user is encouraged to express himself when he wishes, especially when he experiences a strong emotional change.

[0162] According to the invention, a large window F1, for example of duration equal to 1 minute, is applied in a sliding manner, in steps of 3 seconds, starting at the beginning of the video, i.e. 41 extracts per video. In [Fig.l], the signal DF1 represents the data corresponding to the windows F1.

[0163] A short window F2 is also applied, which is a period of interest of 10 seconds upstream of each response 9 and 10 from the user. Within this period of interest, short windows F3 are applied, for example of duration equal to 3 seconds, in steps of 500 ms, starting at the beginning of the window F2, i.e. 15 extracts per window F2, on which the training will make it possible to detect cardiac characteristics.

[0164]

[0165] Once the artificial intelligence model has been trained and configured, it constitutes the emotion model that can be used in operation.

[0166] In [Fig.3] we can see a user 11 in activity permanently receiving the first radio frequency signal 4. The second radio frequency signal 5 coming from the user is captured to undergo processing 6 identical to the processing during training. The difference between the training phase and the exploitation phase is in particular the use of time windows.

[0167] The time windows Fl are applied in a sliding and continuous manner throughout an observation period of the user. Preferably, one is no longer limited to the duration T0 of the training signal.

[0168] The F2 time windows are not used in the operating phase. In fact, the F3 time windows are applied in a sliding and continuous manner.

[0169] Thus, in operation, the time windows F1 and the time windows F3 are applied continuously throughout a user observation period.

[0170] The observation period is not necessarily continuous throughout the day. This observation period can be triggered based on information captured in the environment. For example, only activate the processing and / or the application of time windows on difficult stages of a task being carried out by the user or only when an external alert occurs, for example, an emergency situation.

[0171]

[0172] The data DF1 and DF3 in [Fig. 3] feed an emotion model 12 which is therefore capable of identifying cardiac and respiratory characteristics to enable real-time prediction of an emotion 13. The model 12 can advantageously be derived from the training of the training model 8 in [Fig. 1].

[0173] [Fig.4] is a schematic view illustrating the generation of data training (in training phase) and useful data (in phase operating) from data representative only of cardiac activity according to the invention.

[0174] Indeed, the invention can be implemented solely from data representative of cardiac activity.

[0175] At the end of the digital processing 6, the respiratory activity 6a can be determined and not used or it can simply not be determined at all.

[0176] The cardiac activity 6b is split into two branches to follow parallel processing. On a first branch, the F3 window is applied, with the limits of the F2 windows if we are in the training phase and without limits of the F2 windows when we are in the exploitation phase. The DF3 data resulting from the application of each F3 window feeds the training model or the emotion model.

[0177] On the second branch, we apply the window Fl, with the limit of the duration T0 if we are in the training phase and without limit of the duration T0 when we are in the exploitation phase. The data DF la directly resulting from the application of the window Fl can directly feed the training model 8 or the emotion model 12.

[0178] The invention also provides an embodiment where the data DFla directly originating from the application of the window F1 are first transformed in step 14 into one or more values ​​DFlb before feeding the training model 8 or the emotion model 12.

[0179] The invention further provides an embodiment where the DFla data directly originating from the application of the window F1 and the DFlb data originating from the transformation of the DFla data in step 14 together feed the training model 8 (in the training phase) or the emotion model 12 (in the exploitation phase).

[0180] In step 14, the transformation carried out is the calculation of a value of the variability of the heart rate HRV.

[0181] [Fig. 5] is a schematic view illustrating the generation of training data (in the training phase) and useful data (in the exploitation phase) from data representative only of cardiac activity according to the invention. The same operations are carried out as in [Fig. 4] except here the initial data 6a only concern respiratory activity. Another difference is the transformation step 14 during which at least one of the following values ​​is calculated:

[0182] - respiratory variability (RRV for “Respiratory Rate Variability” in English),

[0183] - a breathing pattern analysis (BPA),

[0184] - respiratory amplitude and its variability (RAV for “Respiratory Amplitude and Variability” in English).

[0185] [Fig.6] is a schematic view illustrating the generation of data training (in the training phase) and useful data (in the operating phase) from data representative of cardiac activity and data representative of respiratory activity according to the invention.

[0186] The same operations are carried out as in [Fig.4] except here the starting data are the respiratory activity data 6a and the cardiac activity data 6b. On one branch we apply the window F1 to the set of data 6a and 6b, and on the other branch we apply the window F3 to the set of data 6a and 6b.

[0187] Thus at transformation step 14, all the values ​​described with reference to figures 4 and 5 can be calculated, but a value of the respiratory sinus arrhythmia (RSA) can also be calculated.

[0188]

[0189] In [Fig.7] is shown an example of a flowchart illustrating steps of a digital processing of the radar signals coming from the user to obtain data representative of the respiratory activity and data representative of the cardiac activity according to the invention. This is processing 6 of [Fig.l].

[0190] Processing starts at step 15.

[0191] In step 16, the FMCW signals are received from the receiving antennas before conversion to intermediate frequency in step 17.

[0192] An analog-to-digital conversion is performed in step 18. Digital signal processing is performed using one or two fixed or floating-point arithmetic real-time digital signal processors that handle several instructions in parallel: interference reduction using DC component suppression (leakage between Tx and Rx), detection of an increase in the noise threshold, reduction of quantization noise due to analog-to-digital conversion, constant false alarm rate (CFAR) detection, windowing for an FFT, and a transform of the signals under a logarithmic scale.

[0193] In step 19, infinite impulse response filtering reduces electronic noise and extracts specific components of the signal. Quadrature conversion is based on decomposition into two orthogonal components: an in-phase component (I) and a quadrature component (Q). This allows both the amplitude and phase of the signal to be analyzed.

[0194] In step 20, a data cube construction of the FMCW signals is carried out through three-dimensional matrices defined by the characteristics of the MIMO radar (number of channels linked to the number of antennas) and the FMCW modulation (time and chirp).

[0195] In step 21, a Range-Doppler-Angle matrix is ​​developed from the data cubes. A succession of 1, 2 and 3 dimensional Fourier transforms is applied for the estimation of the 3 pieces of information: distance, speed and the angle of the target relative to the radar.

[0196] In step 22, a clutter suppression algorithm is implemented. Through a window (sliding in time) of transmission and reception applied to the sought signatures, the static states as well as the slow movements, therefore different from the sought modulations: rib cage, cardiac and respiratory activity, are excluded. The interference between the harmonics and the fundamental frequency of the heart and respiration is improved by the implementation in step 23 of an empirical decomposition algorithm into ensemble modes and the decomposition in step 24 of the signal into an Intrinsic Mode Function (IMF).

[0197] In step 25, a continuous wavelet transform (CWT) is performed in order to improve the identification of the cardiac and respiratory rhythm. This is an adaptive filtering which makes it possible to generate data 6a representative of the respiratory activity and data 6b representative of the cardiac activity.

[0198] The data 6a representative of the respiratory activity undergo normalization in step 26. The data 6b representative of the cardiac activity also undergo normalization in step 27. The normalizations 26 and 27 are optional and may be identical or different.

[0199] In [Fig.8], we can see the user 11 sitting in front of a workstation 28 in a workshop. Transmitting and receiving antennas can be positioned in different positions 29 of the workshop around the user 11. A processing unit 30 can be provided for implementing the method according to the invention. This processing unit can also be directly embedded in a module with the antennas. This processing unit includes the software and hardware means capable in particular of managing the antennas and the signals passing through its antennas. The processing unit is connected by wire or not to the antennas. It is also possible to envisage positioning the processing unit remotely and providing a relay box placed in the workshop to relay the signals between the processing unit and the antennas. Such a processing unit can be high-power to manage several sets of antennas for several users.

[0200]

[0201] The present invention therefore relates to a method for recognizing emotions combining radio frequency radar and artificial intelligence models to derive emotion from the contactless measurement of at least one key physiological signal of at least one person.

[0202] The invention makes it possible to extract the complete cardiac and respiratory signals according to the short and long time windows chosen, but also characteristics classical characteristics of cardiac and respiratory activities such as HRV, RRV, BPA, RAV, RSA and other characteristics within a single heartbeat or in relation to the relationship between different cardiac characteristics of successive heartbeats.

[0203]

[0204] The various embodiments of the present invention comprise various steps. These steps can be implemented by instructions of a machine executable by means of a microprocessor for example.

[0205] Alternatively, these steps may be performed by specific integrated circuits comprising hard-wired logic to perform the steps, or by any combination of programmable components and custom components.

[0206] The present invention may also be provided in the form of a computer program product which may comprise a non-transitory computer memory medium containing instructions executable on a computing machine, which instructions may be used to program a computer (or other electronic device) to perform the method.

[0207]

[0208] Of course, the invention is not limited to the examples which have just been described and numerous adjustments can be made to these examples without departing from the scope of the invention.

[0209]

Claims

Claims

1. Method for generating training data for an emotion model based on artificial intelligence, this method comprising the following steps for a subject: - producing a training signal (3) of duration TO containing emotion-inducing stimuli that the subject receives, - transmitting a first radio frequency signal (4) during the duration TO towards the subject, - receiving a second radio frequency signal (5) containing a physiological signal due to the reflection of said first radio frequency signal on the subject, - digital processing (6) of the second radio frequency signal so as to generate respiratory data (6a) and / or cardiac data (6b), characterized in that it further comprises the following steps: - defining a time window Fl of duration Tl less than TO, - applying several time windows Fl in a sliding manner to the respiratory data and / or the cardiac data;the training data of the emotion model comprising data from each time window F1, - definition of a time window F2 of duration T2 less than T1, and of a window F3 of duration T3 less than T2, - during the emission of the first radio frequency signal, detection of events (9, 10) originating from the subject or calculated on the basis of the behavior of the subject; - at each event, application of several time windows F3 in a sliding manner on the respiratory data and / or the cardiac data of a time window F2 preceding the event; the training data of the emotion model comprising data from each time window F3.;

2. Method according to claim 1, characterized in that the training signal is a virtual reality environment in which the subject is immersed.

3. Method according to claim 1 or 2, characterized in that the training signal comprises one or more stimuli capable of triggering the same stress emotion in the subject.

4. A method according to any preceding claim, ca- characterized in that the training signal comprises several stimuli spaced apart in time, each stimulus being capable of triggering the same predetermined emotion; the stimuli being differentiated by their level of difficulty, complexity, repetitiveness, effort of attention, suddenness or duration.

5. Method according to any one of the preceding claims, characterized in that the data coming from the time window F1 are directly used as training data and / or undergo a calculation of a value of at least one respiratory and / or cardiac biomarker, this value being used as training data of the emotion model.

6. Method according to claim 5, characterized in that the calculation of a value of at least one respiratory and / or cardiac biomarker comprises at least one of the following calculations: - calculation of a heart rate variability (HRV for “heart rate variability” in English), - calculation of a respiratory variability (RRV for “Respiratory Rate Variability” in English) or of a respiratory (BPA for “Breathing Pattern Analysis” in English) or of a respiratory amplitude and its variability (RAV for “Respiratory Amplitude and Variability” in English), - calculation of a respiratory sinus arrhythmia (RSA for “Respiratory Sinus Arrythmia” in English).

7. Method according to any one of the preceding claims, characterized in that the duration Tl is a minimum of 30 seconds with a sliding step of at least 1 millisecond; or a duration Tl of between 30 seconds and 5 minutes with a sliding step of between 1 millisecond and 15 seconds; or a duration Tl of between 30 seconds and 1 minute 15 seconds with a sliding step of between 3 seconds and 5 seconds.

8. Method according to any one of the preceding claims, characterized in that the duration T2 is between 1 second and 30 seconds; T3 being a minimum of 375 milliseconds, or between 375 milliseconds and 30 seconds, or between 1 and 5 seconds, with a step of minimum 1 millisecond.

9. Method according to any one of the preceding claims, characterized in that the first radio frequency signal is a frequency-modulated continuous wave radar signal (FMCW) or a signal ultra wide band ULB (or UWB for “Ultra Wide Band” in English).

10. Method according to any one of the preceding claims, characterized in that the respiratory data and / or the cardiac data undergo a normalization step before application of the time windows.

11. Method according to any one of the preceding claims, characterized in that the emotion model consists of one or more simple or deep learning models.

12. Method according to any one of the preceding claims, characterized in that the emotion model comprises a hybrid recurrent neural network.

13. Method according to claim 12, characterized in that the emotion model comprises a fusion between a closed recurrent unit GRU (for “Gated recurrent unit” in English) and a long-term and short-term memory network LSTM (for “Long Short-term Memory” in English).

14. Method according to any one of the preceding claims, characterized in that the emotion model comprises: - a slow dynamics model dedicated to the data coming from the time windows F1, - a fast dynamics model dedicated to the data from the time windows F3, and - a decision-making algorithm common to the slow dynamics model and to the fast dynamics model.

15. Method according to any one of the preceding claims, characterized in that, in the context of use of an FMCW radar, the transmission of the first radio frequency signal is carried out by means of several transmission antennas and the reception of the second radio frequency signal is carried out by means of several reception antennas according to MIMO technology (for "Multiple-Input Multiple-Output" in English).

16. A computer program product comprising a non-transitory computer memory medium containing instructions executable on a computing machine, these instructions being used to program a computer to execute the method according to any one of the preceding claims.

Images