Method for determining the emotional state of a subject and associated device

Abstract

The invention relates to a computer-implemented method (100) for determining, at an evaluation time (t e ), the emotional state S (t e ) of a person (P) immersed in an environment (E) for the purpose of monitoring the emotional state of said person (P). The invention also relates to an associated determination system (1) and computer program product.

Description

[0001] METHOD FOR DETERMINING AN EMOTIONAL STATE OF A SUBJECT AND ASSOCIATED DEVICE

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to determining an emotional state of a person for monitoring their state of health.

[0004] The present invention relates to a computer-implemented method for determining, at an evaluation time, an emotional state of a subject immersed in an environment for the purpose of monitoring an emotional behavior of this subject and predicting the state of health thereof.

[0005] STATE OF THE ART

[0006] Depressive disorders, or mental health disorders, are common illnesses that affect a large portion of the world's population. Mental health disorders are treatable, but very few patients receive treatment. Mental health disorders generally do not improve on their own, and if left untreated, they can worsen and cause other serious problems. Often, depressed people are unaware of the symptoms of depression, such as apathy, while they are suffering from them.

[0007] Examples of current ways to track people's emotional state include the following.

[0008] For example, monitoring the emotional state of a person placed in an EHPAD (Etablissement d'Hébergement pour Personnes Ancienées Dépendes) or living at home can be carried out on the basis of transmitting information to a doctor (psychiatrist, gerontologist, geriatrician). The information is entered by a care team into a computer system which transmits it to the doctor. The disadvantage of this method lies in the subjectivity of the information entered by the care team and the impact of this subjectivity on the clarity of the information. This results in a difficulty for the doctor receiving the information to monitor the emotional behavior of the person on the basis of this information, in order to prescribe treatment for the person.

[0009] In another example, questionnaires can be carried out through interviews conducted by a psychologist. This type of questionnaire usually includes a reduced amount of information (for example, when it is a questionnaire related to the GDS (Geriatric Depression Scale) - geriatric depression scale, consisting of a limited number of questions that can be reduced to four questions). Furthermore, in general, questionnaires are completed at best every two weeks.

[0010] In yet another example, the physician (GP or psychiatrist) may conduct a clinical interview in which he or she asks the subject questions to determine their emotional state. This type of clinical interview is time-consuming and, in the best case scenario, is conducted at a rate of one interview per month.

[0011] The invention overcomes these drawbacks.

[0012] SUMMARY

[0013] In this context, the invention proposes a solution aimed at monitoring the emotional state of a subject immersed in an environment in order to predict their state of health (i.e., emotional state).

[0014] A first aspect of the invention relates to a computer-implemented method for determining, at an evaluation time, an emotional state of a person immersed in an environment for the purpose of tracking an emotional state of said person, said method comprising the following steps: receiving, during all or part of a time period preceding the evaluation time, at least one first signal acquired by at least one sensor, said first signal being representative of a first time sequence of behavioral and / or physiological attributes of the person during said time period; receiving, during all or part of said time period, at least one second signal acquired by at least one environmental sensor, said second signal being representative of a second time sequence of environmental states during said time period;determining, on the basis of the at least one first signal, a third temporal sequence of emotional signatures of the subject during said time period, said third sequence forming a first set of emotional signatures of the subject, each of the emotional signatures of the person of the third sequence being represented by a position in a space representative of the emotions; determining, on the basis of the at least one second signal, a fourth temporal sequence of signatures of the environment during said time period, said fourth sequence forming a second set of signatures of the environment, each of the signatures of the environment being represented by a position in said space representative of the emotions; determining a degree of similarity between the first set of emotional signatures of the subject and the second set of signatures of the environment;determine the emotional state of the person at the time of assessment from the degree of similarity determined.;

[0015] Thus, the determination method presented here allows, by comparison between the emotional signatures of a person over a given period and the signatures of the environment over the same given period, to estimate an emotional contagion from the environment of the person towards the latter, and to estimate the reactivity of the person to his environment. For example, this makes it possible to estimate whether the person is suffering from apathy.

[0016] In some embodiments, determining the degree of similarity between the first set of emotional signatures of the person and the second set of signatures of the environment comprises calculating a distance in said representative space between the first set of emotional signatures of the person and the second set of signatures of the environment. In some embodiments, the distance is calculated using a cosine similarity between a first vector representative of the first set of emotional signatures of the person and the second set of signatures of the environment.

[0017] In some embodiments, the calculation of the distance comprises a weighting depending on the positions of the emotional signatures of the subject of the first set and the signatures of the environment of the second set in said representative space.

[0018] In some embodiments, the calculation of said distance comprises a weighting dependent on time instants corresponding to the emotional signatures of the person in the third time sequence and to the signatures of the environment in the fourth time sequence.

[0019] In some embodiments, the calculation of said distance is performed between the emotional signatures of the person of a first sub-sequence of the third temporal sequence and the signatures of the environment of a second sub-sequence of the fourth temporal sequence, said first sub-sequence and second sub-sequence corresponding to a temporal sub-sequence of said temporal period.

[0020] In some embodiments:

[0021] - at least one first signal is acquired by at least one sensor chosen from: a camera, a webcam, a three-dimensional camera, a microphone, an inertial unit, a physiological sensor, a headset dedicated to electroencephalographic measurements,

[0022] - the method is configured to extract from the at least one first signal the behavioral and / or physiological attributes in the first time sequence so as to obtain values ​​representative of at least one of the following attributes among: a gesture of the person, a movement of the person, an expression of the face of the person, a movement of the eyes of the person, physiological parameters of the person, an interaction with an electronic device.

[0023] In some embodiments:

[0024] - the at least one second signal is acquired by the at least one environmental sensor chosen from: a camera, a webcam, a three-dimensional camera, a microphone, a means of accessing a remote server storing information relating to one or more audiovisual, visual, or audio contents broadcast during all or part of the time period, an electromagnetic detector, and in that the method is configured to extract from the at least second signal the state of the environment, so as to obtain values ​​representative of a state chosen from: a quantity of movement in an audiovisual or visual content, broadcast during all or part of said time period, a sound level and / or a frequency spectrum of an audiovisual or audio content broadcast during all or part of said time period, a sound level and / or a sound frequency spectrum in a close vicinity of the subject,an electromagnetic signal representative of a variation in an electromagnetic signature of a neighborhood close to the subject.,

[0025] A second aspect of the invention relates to a system for determining, at an evaluation time, an emotional state of a person immersed in an environment for the purpose of monitoring a state of health of said person, comprising at least one sensor, at least one environmental sensor, and a programmable device, said programmable device being adapted to: receive, during all or part of a time period preceding the evaluation time, at least a first signal acquired by F at least one sensor, said first signal being representative of a first time sequence of behavioral and / or physiological attributes of the subject during said time period; receive, during all or part of said time period, at least a second signal acquired by F at least one environmental sensor, said second signal being representative of a second time sequence of a state of the environment during said time period;determining, on the basis of the at least one first signal, a third temporal sequence of emotional signatures of the person during said time period, said third sequence forming a first set of emotional signatures of the person, each of the emotional signatures of the person of the third sequence being represented by a position in a space representative of a panel of emotions; determining, on the basis of the at least one second current signal, a fourth temporal sequence of signatures of the environment during said time period, said fourth sequence forming a second set of signatures of the environment, each of the signatures of the environment being represented by a position in said representative space;determining a degree of similarity between the first set of emotional signatures of the subject and the second set of signatures of the environment, determining the emotional state of the person at the time of evaluation from the determined degree of similarity.;

[0026] A third aspect of the invention relates to a computer program comprising instructions for implementing the following steps of a method for determining, at an evaluation time, an emotional state of a person immersed in an environment for the purpose of monitoring a state of health of said person during the execution of the program by a processor of a programmable device: receiving, during all or part of a time period preceding the evaluation time, at least a first signal acquired by at least one sensor, said first signal being representative of a first time sequence of behavioral and / or physiological attributes of the subject during said time period;receiving, during all or part of said time period, at least one second signal acquired by at least one environmental sensor, said second signal being representative of a second time sequence of a state of the environment during said time period; determining, on the basis of the at least one first signal, a third time sequence of emotional signatures of the person during said time period, said third sequence forming a first set of emotional signatures of the person, each of the emotional signatures of the person of the third sequence being represented by a position in a space representative of a panel of emotions; determining, on the basis of the at least one second current signal, a fourth time sequence of signatures of the environment during said time period, said fourth sequence forming a second set of signatures of the environment;determining a degree of similarity between the first set of emotional signatures of the subject and the second set of signatures of the environment; determining the emotional cognitive state of the person at the time of evaluation from the determined degree of similarity.;

[0027] The present disclosure also relates to a non-transitory computer-readable recording medium comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any of the embodiments.

[0028] Such a non-transitory computer-readable recording medium may be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor device, or any suitable combination thereof. It should be noted that the following examples, while more specific, are merely an illustrative and non-exhaustive list, readily appreciated by a person of ordinary skill in the art: a portable computer floppy disk, a hard disk drive, a ROM, an EPROM (Erasable Programmable ROM) or Flash memory, a portable CD-ROM (Compact-Disc ROM).

[0029] DEFINITIONS

[0030] In the present invention, the terms below are defined as follows: The expression "Emotional cognitive state of a person" designates a state of mind such as a sensation, an emotion, a feeling, a mood resulting from a set of psychic phenomena which influence both the person's own state of mind, attitude, worldview, thought, and behavior in the world.

[0031] The expression "negative state" of a person designates a state of mind generating psychological suffering in the person.

[0032] The term "Emotional Signature" refers to a representation of an emotional state in a space representative of emotions.

[0033] The expression "Environment of a person at a given moment" means, within the framework of this description, the set of physical elements with which the person interacts or likely to induce a reaction from the person. For example, the environment may include other people interacting with the person analyzed, a robot such as a humanoid, a connected speaker or a chatbot, a sound and / or visual environment, such as a telephone, a laptop, a television or more broadly sources of information broadcasting signals and perceptible to the person at the given moment.

[0034] The term "environmental state" refers, as used herein, to a characterization of sensory stimuli present in the environment that may alter a person's emotional state.

[0035] The expression "environmental signature" designates, within the framework of this description, a representation in a space representative of the emotions of a state that the environment would be likely to diffuse and therefore to be perceived by a person located in this environment.

[0036] "Apathy" is defined as a person's indolence or indifference, even to the point of complete insensitivity.

[0037] "Emotional tone" refers, in the context of this description, to the set of emotional states that person P experiences during a given period. Thus, emotional tone allows us to infer dynamism, vigor, and energy assessed through the amplitude of emotions produced by a subject.

[0038] The term "heat map" refers to a graphical representation of an accumulation of emotions, in the form of an image represented by a two-dimensional matrix of pixels, each pixel representing an emotion. The intensity of each pixel in the heat map is related to the number of times the emotion associated with the pixel has been identified over the period in which the accumulation is calculated.

[0039] DESCRIPTION OF FIGURES

[0040] Figure 1 is an example of steps implemented for carrying out the method of determining an emotional state of a subject, according to one or more embodiments.

[0041] Figure 2 represents an example of a temporal sequence of emotional signatures of a person represented in a representative space of emotions.

[0042] Figure 3 shows an example of an emotion heat map representing a person's emotional tone in an emotion space. Figure 4 shows an example of a temporal sequence of environmental signatures represented in an emotion space.

[0043] Figures 5a and 5b represent two emotion heat maps of a healthy person at two different assessment times within a given assessment period. Figure 5c represents an emotion heat map of the same person determined at the end of the given assessment period.

[0044] Figures 6a, 6b, and 6c represent three emotion heat maps of a person suffering from depression, at three different assessment times within a given assessment period. Figure 6d represents an emotion heat map of the same person, determined at the end of the given assessment period.

[0045] Figure 7 shows an emotion heat map of a given person during a given assessment period, on the left, and an emotion heat map produced by the person's environment, during the given assessment period, on the right.

[0046] Figure 8 is a schematic diagram showing some of the components of an example of a determination system configured to implement the method of determining an emotional state of a subject.

[0047] Figure 9 is a representative curve of the evolution of the emotional tone of a person entering an EHPAD, where the emotional tone is determined by the determination method described here.

[0048] DETAILED DESCRIPTION

[0049] A first aspect of the invention relates to a computer-implemented method 100 for determining, at an evaluation time t e , an emotional state S(t e) of a person P immersed in an environment E.

[0050] We wish in fact, by determining and monitoring the evolution of the emotional state S(t) of the person P between an instant ti preceding the instant of evaluation t e and the evaluation time t e , to be able to detect negative states, or to detect a change in the person's emotional state towards a negative state, in order to monitor their state of health. A particular application lies in the detection of warning signs of depression in a person. In this application, it is advantageous to determine the person's emotional cognitive state at different times tk, where k is an integer between 1 and K.

[0051] The method 100 is for example implemented by a system 1 for determining an emotional state of a person P immersed in an environment E. As will be explained later, the method 100 uses a plurality of signals captured between an initial instant ti and the evaluation instant t e , to determine the emotional state of person P at the evaluation time t e .

[0052] Advantageously, the determination system 1 comprises a plurality of sensors CPi intended for monitoring the person P, a plurality of environmental sensors CEj and a programmable device, i and j being integers between 1 and I, and 1 and J respectively.

[0053] A programmable device means any programmable information processing system such as a processor, a business computer, a personal computer (PC), a smartphone or a tablet.

[0054] The plurality of CPi sensors is intended to acquire a plurality of first signals Spi(t) during the time period [ti,t e ]. Each of the first signals Spi(t) is representative of a first temporal sequence relating to a behavioral and / or physiological attribute of the person P. A behavioral attribute of a person is understood to be an attribute relating to their behavior, such as their body movements or their facial expression. A physiological attribute of a person is understood to be an attribute relating to their physiology, such as their heart rate, brain activity, or respiratory rate. The first signals Spi(t) can be discrete or continuous. They are also either scattered during the time period [ti,t e ], either recorded according to an ordered schedule during the time period [ti,t e]. For example, the first Spi(t) signals can be recorded initially once a week, then once a day, then every second for one hour each day.

[0055] The plurality of environmental sensors CEi is intended to acquire a plurality of second signals Sej(t) during the time period [ti,t e ]. Each of the second signals Sej(t) is representative of a second temporal sequence of states of the environment E. The second signals Se t) can be discrete or continuous. They are also either scattered during the time period [ti,t e ], either recorded according to an ordered schedule during the time period [ti,t e ]. For example, the second Sej(t) signals can be recorded initially once a week, then once a day, then every second for one hour each day.

[0056] For example, the plurality of sensors CPi comprises a sensor Ci consisting of a camera, or a webcam, or a three-dimensional camera filming at least one part of the body of the person P. The at least one part of the body comprises the face of the person P. In this case, the first signal Spi(t) is representative of a change in the expression of the face of the person P, the movement of his eyes or the movements of his head over time.

[0057] Typically, the movement of reference points on the images captured by the camera, such as points located on the outline of the person P's mouth, or on their eyebrows, is analyzed in order to profile the person's emotion. This may consist, for example, of identifying a smile on the face of the person P, or a frown.

[0058] Detecting points representative of an expression (those characterizing the outline of the eyes and mouth for example) can be achieved by Machine Learning algorithms such as Random Eorest or Deep Learning algorithms such as convolutional neural networks. These tools can also be used to recognize facial expressions among six expressions such as Ekman expressions, without analyzing points representative of the face, but directly from the pixels of the image representing the face. These techniques have become common and are integrated into smartphones to automatically trigger the shot when the person photographed smiles.

[0059] For example, the plurality of sensors CPi comprises a sensor C2 consisting of a microphone capable of recording the voice of the person P. In this case, the first signal Sp2(t) is representative of an evolution of the tone and / or amplitude of the voice of the person P. In the same way that a classifier can analyze the expression of the face, it is possible to rely on characteristics of the audio signal Spi(t) such as the cepstrums or the formants to identify the emotion contained in the voice by means of classifiers such as autoencoders or neural networks.

[0060] For example, the plurality of sensors CPi comprises a sensor C3 consisting of an inertial unit worn by the person P, for example embedded in a connected object worn by the person P such as a connected watch, and capable of recording their position and movements. In this case, the first signal Sps(t) is representative of an evolution of the gestures of the person P and the movements of their body. It is possible to deduce from the first signal Sps(t) an estimate of the pose (the skeleton characterized by a set of points generally corresponding to the joints) of the person. The analysis of emotions from the pose then consists of analyzing the movement of the person over time to deduce the emotion. Tools from the field of deep learning such as LSTM can be used, as well as other tools belonging to the field of machine learning and dedicated to the recognition of temporal patterns.

[0061] For example, the plurality of CPi sensors includes a physiological sensor capable of detecting physiological signals of the person P. For example, the physiological sensor may be a sweating or heart rate sensor. For example, a high heart rate will allow one to conclude that the person P is stressed or extremely agitated. Commercial sensors may be used (such as Biopac MP150, Empatica E4, etc.), the signals of which may be analyzed by machine learning.

[0062] For example, the plurality of CPi sensors includes a headset dedicated to electroencephalographic measurements, in order to detect the brain activity of the person P, such as their level of concentration or their state of stress.

[0063] For example, the plurality of CPi sensors comprises a sensor capable of measuring the presence of a person other than the person P in a room, such as a motion sensor, a magnetic or electromagnetic fingerprint detector, or an infrared detector. For example, the plurality of CPi environmental sensors comprises an electromagnetic detector such as a fall detection system using RADAR technology, such as the Morphee+™ product. Such a sensor makes it possible to analyze the movement of the person in their environment. In the same way as other conventional motion sensors, such a sensor therefore makes it possible to analyze the movement of the person and associate it with emotions. For example, a rapid movement (associated with a positive emotion) when the person enters their room will be easily differentiated from a slow movement which will reflect withdrawal, a lack of vitality very characteristic of depression.

[0064] For example, the plurality of environmental sensors CEj comprises an environmental sensor CEi consisting of a camera, a webcam, a three-dimensional camera capable of filming the location where the person P is located, for example, a room. In this case, the second signal Sei(t) may be representative of the presence or absence of other people in the room.

[0065] For example, the plurality of environmental sensors CEj comprises an environmental sensor CE2 consisting of a microphone. In this case, the second signal Se2(t) may be representative of the sounds emitted in the environment where the person P is located. The difficulty then consists in separating the audio sources of the environment to identify their origin. For example, recurrent variational autoencoders used in the field of deep learning make it possible to separate the vocal audio sources. It is then possible to identify the voice of the person P and then to focus, for example, on another voice existing in the environment if, for example, another person P' is in conversation with the person P. The audio emotion recognition tools are then applied to recognize the emotion contained in the voice of the person P' which constitutes the emotion produced in the environment of the person P.

[0066] For example, the plurality of environmental sensors CEj comprises an environmental sensor CE3 consisting of a means for accessing a remote server storing information relating to one or more audiovisual, visual, or audio contents broadcast during all or part of the time period [ti,t e ]. Such a means of access may, for example, be an Internet connection allowing access to the description of an audiovisual program currently being played by person P (comedy, drama, etc.). For example, if the program currently being played is a comedy, the emotional content of the environment will present joyful and positive states. Conversely, if the program is a drama, the emotional content of the environment will present distressing and stressful states.

[0067] Generally, some of the CEj environment sensors may coincide with some of the Ci sensors.

[0068] The method of determining 100 an emotional state of the person P, at the evaluation time t e , will now be described. Figure 1 illustrates the main steps carried out during the implementation of the determination method 100.

[0069] In a reception step relating to the person P, ElOOa, all of the first signals Spi(t) are received by the different sensors CPi over the time period [ti,t e ].

[0070] In a reception step relating to the environment E, ElOOb, all of the second signals Sej(t) are received by the different environmental sensors CEj during the time period [ti,t e ] .

[0071] Typically, steps ElOOa and ElOOb are concomitant. Indeed, they both take place during the time period [ti,t e ].

[0072] In a determination step E200a relating to the person P and subsequent to steps El00a and El00b, a third temporal sequence of emotional signatures of the person P is determined on the basis of the set of first signals Spi(t). The set of emotional signatures of the third temporal sequence forms a first set. Each emotional signature is represented by a position in a space representative of the emotions R. For example, a first signal Spi(t) corresponding to images of the face of the person P, and therefore comprising representations of the expressions of his or her face, will be analyzed to determine the corresponding emotional signatures.For example, we consider five CPi sensors, such as a webcam recording images captured at 25Hz, a microphone recording a sound sampled at 48KHz, an infrared sensor recording an infrared image sampled at 50Hz, a bracelet recording sweating at the wrist at a sampling frequency of 100Hz and heart rate at a sampling frequency of 100KHz. Each of the recorded signals is preprocessed appropriately as exemplified previously, namely: by convolutional neural networks to process the images, formants to analyze the sound, derivatives for sweating and heart rate. Individually, each preprocessing result is used to perform the regression operation which consists, for each CPi sensor, in analyzing the preprocessing result to estimate the emotion felt by the person.

[0073] As an example of implementation, an artificial neural network that has as input the images of the person's face will produce as output the coordinates in the Valence Arousal space of the emotion contained in these images. This network will have been trained beforehand in supervised or unsupervised mode, to produce such an output from the facial images. In the same way, the speech produced by the person can be processed to provide a spectrogram whose energy maxima will represent the formants. All of these formants during the period considered [ti,t e ] can be projected into a reduced space (or variety) allowing the emotion contained in the vocal signal to be estimated.

[0074] More generally, the signals recorded by each of the CPi sensors are used to estimate the emotion felt by the person P using signal processing tools such as machine learning or deep learning.

[0075] Thus, five emotions are estimated from the five pre-processed signals and the estimated emotion associated with each of the pre-processed signals. The fusion of the five emotions is carried out, for example, by an average to identify the final emotion of the person in the Valence Arousal space.

[0076] The representative space of emotions R can be, as non-limiting examples: a discrete space of emotions, for example: a binary measure of positive / neutral, negative / positive or positive / negative emotion, or a discrete set of n emotions, such as the six emotions of Ekman,

[0077] - a continuous space of emotions: for example, the valence / arousal space, a latent space characterized by a principal component analysis, by a generative adversarial network (GAN), or by an auto-encoder.

[0078] Figure 2 shows an example of a representation in valence / arousal space of a third temporal sequence of emotional signatures of a person P. Valence corresponds to the negative (leftward) or positive (rightward) aspect of the emotional signature. Arousal corresponds to the passive (downward) or active (upward) aspect of the emotion. The space around the origin (intersection of the horizontal and vertical axes) is the seat of neutral emotions. The further one moves away from the origin towards the periphery, the greater the intensity of the emotion. Each black sampling point (represented by a filled circle) represents an emotional signature at a time tk. The set of chronologically ordered emotional signatures forms a trajectory.

[0079] A representation of the cumulative trajectory points defined by the set of sample points consists of an emotion heat map that can be constructed based on the set of emotional signatures of person P determined during the time period [ti,t e ]. During this time period, a histogram is produced of all the emotional signatures determined at different times during the time period [ti,t e ]. The occurrence level of emotional signatures is visualized as a color level of the pixels corresponding to each emotional signature in the two-dimensional matrix representing the valence / arousal space.

[0080] Figure 3 gives an example of such a heat map, in the valence / arousal space. The black pixels correspond to a low frequency, or low accumulation of the corresponding emotional signature and the light pixels correspond to a high frequency, or high accumulation of the corresponding emotional signature. In other words, the light pixels correspond to a high accumulation of the same emotional signature. In a determination step E200b relating to the environment E, subsequent to steps El00a and El00b, a fourth sequence of signatures of the environment E is determined on the basis of the set of second signals Sej(t). The set of signatures of the environment of the fourth temporal sequence forms a second set.Each environmental signature is represented by a position in the emotion representative space R, where the emotional signatures of the person P during the time period [ti,t] are represented. e ].

[0081] Figure 4 shows an example of representation in the valence / arousal space of a fourth temporal sequence of signatures of an environment E. The set of signatures ordered chronologically forms a trajectory T (represented by a broken line).

[0082] As with the emotional signatures of person P, it is possible to construct a heat map of states of the environment E.

[0083] As will be described later, the determination method 100 presented herein makes it possible to detect warning signs of depression in a person.

[0084] One aspect of the invention is based on the fact that if person P is in good health, they will be reactive to the environment E. In other words, their emotional state will be close to the state of the environment E. This proximity can be called "emotional contagion". On the contrary, if the state of health of person P deteriorates, they will be less and less reactive to the environment E. In other words, their emotional state will move away from the state of the environment E.

[0085] An interesting factor to observe is then the emotional tone of a person P. By emotional tone, we mean the set of emotional states through which the person P passes during a given period. A healthy subject passes through all emotional states. Conversely, the dynamics of emotions of a person beginning to sink into depression gradually decreases. In other words, the emotional tone of this person narrows around a zone of neutral emotions. In the valence / arousal space, the zone of neutral emotions is located around the intersection of the valence axis and the arousal axis, that is, in the center of the space. The emotional tone can be estimated on the basis of the surface in the valence / arousal space of the area with bright pixels covering the space. The wider this zone is, the more satisfactory the emotional tone of the person is. The narrower this zone is, the more degraded the emotional tone of the person is.

[0086] Figures 5a and 5b show two emotion heat maps, in the valence / arousal space, of a person with an intermediate emotional tone at two successive times, respectively 19 days before the assessment time t e and 10 days before the evaluation time t e Figure 5c shows the emotion heat map of the same person over the time period [ti,t e ]. It can be observed that the light pixels in Figure 5c cover a major part of the valence / arousal space. The emotional tone associated with Figure 5c is good.

[0087] Figures 6a, 6b and 6c show three heat maps of emotions of a person in depression, at different successive moments, respectively 27 days before the assessment moment t e , 17 days before the evaluation time t e and 9 days before the evaluation time t eFigure 6d shows the heat map of emotions of the same person over the time period [ti ,t e ] . We can observe in Figure 6a that the emotional tone of the person is good, in other words, that the latter starts from a good state of health. We can observe in Figure 6b that at this moment of the evaluation, the emotional tone of the person is intermediate, in comparison to the observation in Figure 6a. We can observe that the light pixels in Figures 6c and 6d only cover a reduced portion, centered around the neutral emotions zone, of the valence / arousal space, which reflects a low emotional tone.

[0088] Thus, one of the advantageous applications of the method 100 for determining an emotional state described here is the detection of a symptom of the development of apathy in the person P. Apathy is defined as someone's indolence or indifference, pushed to the point of complete insensitivity. Apathy is a precursor symptom of depression. The determination method 100 described here thus makes it possible, as already mentioned, to detect warning signs of this pathology. Thus, in a comparison step E300 subsequent to steps E200a and E200b, the determination system 1 determines a degree of similarity between the first set and the second set. As a reminder, the first set is the set of emotional signatures of the third time sequence determined during step E200a and the second set is the set of environmental signatures of the fourth time sequence determined during step E200b.

[0089] For example, during the comparison step E300, an emotion heat map Mi of person P and a state heat map M2 of his environment E can be compared, at the evaluation time t e . The degree of similarity can then be a distance between the emotion heat map Mi and the state heat map M2. The distance is compared to a threshold value.

[0090] Figure 7 shows on the left an emotion heat map Mi of person P and on the right a state heat map M2 of his environment E corresponding to the accumulation of measurements during a period Pe. The emotional signatures corresponding to the map Mi come from the analysis of the emotions expressed on the face of person P. The environment signatures corresponding to the map M2 come from the analysis of the sound content of a television program viewed by person P.

[0091] In Figure 7, it can be observed that the emotional content that the television program is likely to transmit to the subject is emotionally varied, ranging from passive emotions at the bottom to positive and active manifestations at the top right of the M2 map. This emotional content can be quantified by the area occupied by the light pixels on the heat map. The emotions expressed by person P exposed to the audiovisual stimuli of the television program remain concentrated around the region associated with neutral emotions, as visible on the Mi map. A preliminary conclusion can be that person P is not in unison with the television program.

[0092] If the distance is less than the threshold value, the emotion heat map Mi and the state heat map M2 are considered close. In other words, the method 100 determines that there has been emotional contagion from the environment E to the person P.

[0093] If the distance is greater than the threshold value, the method 100 determines that there is no longer emotional contagion from the environment E to the person P. In particular, it is deduced that the person P has become apathetic.

[0094] The distance between the emotion heat map Mi and the state heat map M2 is advantageously a distance defined in a vector space (such as the Euclidean distance, the Mahalanobis distance).

[0095] Alternatively, the degree of similarity determined in step E300 is a similarity measure determined by an artificial neural network, or other techniques known in the field of image processing or machine learning.

[0096] Advantageously, in the case where the degree of similarity determined in step E300 is a distance calculated between an emotion heat map Mi of the person P and a state heat map M2 of his environment E, this degree of similarity can be a function of several parameters. It is assumed that the arousal valence space is a set of N pixels, where each pixel corresponds to an emotion. The emotion heat map Mi and the state heat map M2 each consist of a set of N numerical values. Each numerical value of the map Mi or the map M2 is representative of the occurrence of the emotion associated with the corresponding pixel during the period [ti,t e ].

[0097] For example, a first parameter PixelSet corresponds to the set of data used to calculate the distance d. For example, the PixelSet parameter corresponds to the set of N pixels in the valence / arousal space.

[0098] For example, a second Filter parameter is a set of coefficients each assigned to a pixel of the set of N pixels in the valence / arousal space. Each coefficient can be chosen to favor the importance of the emotion corresponding to the pixel to which the coefficient is assigned in the calculation of the distance.

[0099] The second Filter parameter can for example be represented by the binary image of a black disk (black = 0) of radius 0.5 (if the Valence Arousal space is represented between -1 and 1 on each axis) on a white background (white = 1), this disk being centered at [0,0] in the Valence Arousal space. Taking it into account will allow to consider only emotions having a certain amplitude: between the edges of this disk of radius 0.5 and the Valence Arousal disk of radius 1. Thus neutral emotions will not be taken into account in the calculation of the distance but only significant emotions close to the periphery.

[0100] For example, a third parameter TimeFilter is a set of coefficients assigned to the measurement times t p of the first signals Spi(t) and the second signals Ssj(t) during the period [ti,t e ] in order to promote the importance of certain measurement moments t p in the calculation of distance. For example, significant coefficients can be assigned to the measurement times t p closest to the evaluation time t e , to promote the importance of the last emotional signatures of the person P and the last signatures of his environment E.

[0101] For example, a fourth parameter, Latency, makes it possible to take into account the latency time that may exist between the appearance of a state of the environment and the effect of this appearance on the emotional state of the person P. For example, the Latency parameter is a vector comprising the values ​​of a discretized Heaviside function whose transition between the value 0 and the value 1 is a function of the latency time considered.

[0102] In an advantageous embodiment, the degree of similarity determined in step E300 is a cosine similarity between a first vector formed of the N numerical values ​​corresponding to the heat map of emotions Mi and a second vector formed of the N numerical values ​​corresponding to the heat map of states M2 of the environment E. The cosine similarity Sim between two vectors A and B is defined by the scalar product of these two vectors divided by the product of their norm:

[0103] Sim=AB / (|| A ||*|| B ||) .

[0104] In this case, the degree of similarity is between 1 and -1. When the degree of similarity is close to 1, there is a high consistency between the first vector and the second vector, and they are considered close, while when the degree of similarity is close to -1, the first vector and the second vector are considered distant.

[0105] Another interesting factor to observe is the synchronization of the breaks in emotional behavior between the evolution of the emotional states of the person P and the evolution of the states of his environment E. For example, if the environment emits a particular emotion Eel(t) at a time t and a second later a particular emotion Epl(t) is captured from the person P, even if the emotion Epl(t) is different from the state Eel(t) of the environment, then the emotional changes between the person and his environment are synchronized, which means that the person is sensitive to the emotional character of the environment E. The observation of this factor allows us to conclude that the person is not apathetic, because of his change in emotional state following the emission of a particular state of his environment.

[0106] Advantageously, the determination method 100 is renewed periodically to monitor at different successive instants tk in time, during a period [ti,tf], the evolution of the emotional contagion of the environment E towards the person P during the period [ti,tf].

[0107] In one embodiment, the first Spi(t) signals are captured continuously during the period [ti,tf]. For an implementation of the determination method 100 at a time tk included in the period [ti,tf], only the acquired values ​​of these first Spi(t) signals during the period [ti,tk] are used during steps E200a, E200b and E300.

[0108] Similarly, in this embodiment, the second signals Sej(t) are captured continuously during the period [ti,tf]. For an implementation of the determination method 100 at a time tk included in the period [ti,tf], only the acquired values ​​of these second signals Ssj(t) during the period [ti,tk] are used during the steps E200a, E200b and E300;

[0109] In another embodiment, only the values ​​of the first signals Spi(t) and the second signals Ssj(t) included in a window [t e -T,t e ] of duration T are taken into account during steps E200a, E200b and E300. For example, the duration T is 7 days. If the duration T corresponds to the period of time between two determinations of emotional signatures of the person P and between two determinations of signatures of the environment E, the degree of similarity determined in step E300 is an instantaneous degree of similarity.

[0110] Advantageously, it takes place before a first evaluation moment t e , a calibration period during which the determination method 100 is implemented several times in order to determine a degree of similarity typical of the person P. Thus, from the first evaluation time t e , the degree of similarity determined in step 300 of the determination method 100 will be compared to the typical degree of similarity of the person P, so as to be able to estimate whether the state of health of the person P is good compared to his usual state of health, or whether the person P tends towards a negative state.

[0111] One of the advantages of the invention is that the determination method is carried out on the basis of objectively captured signals. The determination method has a high reproducibility due to this objective assessment, unlike clinical studies carried out on the basis of questionnaires completed subjectively either by qualified persons, or by analyzed subjects, or by operators with more or less experience.

[0112] A second aspect of the invention relates to the determination system 1.

[0113] The determination system 1 is configured to implement the method, previously described, of determining an emotional state of a person P, to monitor their state of health.

[0114] Figure 8 is a schematic diagram showing some of the components of an example of the determination system 1.

[0115] The determination system 1 may be implemented as a single hardware device, for example, as a desktop personal computer (PC), a laptop, a personal digital assistant (PDA), a smartphone, a server, a console, or may be implemented on separate hardware devices interconnected by one or more communication links, with wired and / or wireless segments. The determination system 1 may, for example, be in communication with one or more cloud computing systems (i.e., on the cloud), one or more remote servers or devices to implement the functions described herein for the device in question. The determination system 1 may also be implemented itself as a cloud computing system.

[0116] As shown schematically in Figure 8, the determination system 1 may comprise at least one processor 10 configured to access at least one memory 20 comprising computer program code 70. The computer program code may comprise instructions configured to cause the determination system 1 to execute one or more or all of the steps of the determination method 100 described previously.

[0117] The processor 10 may thus be configured to store, read, load, interpret, execute and / or otherwise process the computer program code 70 stored in the memory 20 such that, when the instructions encoded in the computer program code are executed by the at least one processor 10, the system 1 executes one or more steps of the determination method 100 described herein.

[0118] The processor 10 may be any suitable microprocessor, microcontroller, integrated circuit, central processing unit (CPU), graphics processing unit (GPU), or tensor processing unit (TPU) including at least one hardware-based processor or processing core.

[0119] The memory 20 may comprise random access memory (RAM), cache memory, non-volatile memory, backup memory (e.g., programmable or flash memories), read only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), or any combination thereof. The ROM of the memory 20 may be configured to store, among other things, an operating system of the system S and / or one or more computer program codes of one or more software applications. The RAM of the memory 20 may be used by the processor 10 for temporary storage of data. The system 1 may further comprise one or more communication interfaces 40 (e.g., network interfaces for access to a wired / wireless network, including an Ethernet interface, a WIFI interface, USB interfaces, etc.The determination system 1 may include other associated hardware such as user interfaces such as a 2D visual rendering interface 30A, in our case containing a 3D screen and 3D glasses, or haptic interfaces 30B, or any other interfaces (e.g. keyboard, mouse, display screen, etc.) connected via one or more suitable communication interfaces 40 with the processor. The determination system 1 may also include a media reader 50 for reading a computer-readable external storage medium. The processor 10 is connected to each of the other components in order to control their operation.

[0120] A third aspect of the invention relates to a computer program product comprising instructions for implementing the steps of the method for determining, at an evaluation time, an emotional state of a subject immersed in an environment for the purpose of monitoring a state of health of said subject during execution of the program by a processor of a programmable device. For example, the instructions are part of the computer program code 70 stored in the memory 20 of the determination system 1.

[0121] The computer program product notably implements steps E200a, E200b, E300.

[0122] EXAMPLES

[0123] The present invention will be better understood by reading the following examples which illustrate the invention in a non-limiting manner.

[0124] Example 1:

[0125] In this example, person P is an elderly person in their room whose health status is to be monitored based on their emotional state. Person P is facing their television. Environment E mainly consists of the program broadcast on television.

[0126] Determination system 1 includes a small robot with a webcam and connected to the Internet.

[0127] The sensor Ci therefore corresponds to the image sensor of the webcam, turned towards the face of the person P.

[0128] The CEi environmental sensor is the webcam microphone that records sound from the program broadcast on television.

[0129] The small robot comprises an embedded system with a programmable device capable of analyzing the images recorded by the sensor Ci, in other words capable of performing step E200a of the determination method 100. Typically, analyzing the facial expressions of the elderly person P in the recorded images makes it possible to determine the corresponding emotional signatures. A method such as that described in “Estimation of continuous valence and arousal levels from faces in naturalistic conditions”, Toisoul et al., Nature Machine Intelligence volume 3 (2021) can be used.

[0130] The programmable device also analyzes the sound recorded by the CEi environment sensor, i.e., it performs step E200b of the determination method 100. Typically, the analysis of the recorded sound makes it possible to determine the corresponding environmental signatures. A method such as that described in “Dawn of the transformer era in speech emotion recognition: closing the valence gap”, Wagner (2022) can be used.

[0131] These in situ analyses ensure data confidentiality since the images are never recorded on physical memory but processed in real time in the acquisition flow.

[0132] The small robot sends, following steps E200a and E200b, the emotional signatures of the person P and the signatures of the environment E. Step E300 of determining the degree of similarity is carried out in the cloud and the degree of similarity determined can be received by health professionals following the elderly person P.

[0133] Example 2:

[0134] This example illustrates a use of the determination method 1 and the determination system 1 previously described in the context of monitoring and care of a person P arriving in an EHPAD.

[0135] It is assumed that person P is followed by a team of health professionals and that the determination method 100 is implemented regularly with a determination system 1.

[0136] The team of health professionals monitors, at each iteration of step E200a of determination relating to person P, the corresponding emotional tone on the basis of the set of emotional signatures of person P, also previously denoted first set. For example, and as explained above, the emotional tone can be quantified by the total area occupied in the valence / arousal space by the first set. The higher the emotional tone, the healthier person P is. The lower the emotional tone, the more person P tends towards a negative state.

[0137] Thus, if a drop in emotional tone is noted, the team of health professionals can initiate a range of care for person P, while monitoring the evolution of their emotional tone.

[0138] Figure 9 illustrates the evolution of the emotional tone of person P. We observe that after one week, person P begins to feel depressed. This is in fact a symptomatic event of entering the institution. Depression is diagnosed two weeks later, based on the observation of warning signs of depression. If assistance is provided to person P, such as advice aimed at encouraging them to bond with residents of the institution, we can observe a regrowth of the emotional tone of the person being followed, as visible in Figure 9. Reference signs: t e : evaluation moment

[0139] St e ): emotional state of a person P at time t e

[0140] CPi, i between 1 and I: sensors

[0141] E: environment

[0142] CEj, j between 1 and J: environmental sensors

[0143] Spi(t), i between 1 and I: first signals

[0144] Sej(t), j between 1 and J: second signals

[0145] R: space representing emotions

[0146] Ml: Emotion Heat Map

[0147] M2: Heat map of environmental states

Claims

CLAIMS 1. Determination system (1), at an evaluation time (t e ), of an emotional state S (t e ) of a person (P) immersed in an environment (E) for the purpose of monitoring an emotional state of said person (P), comprising at least one sensor (CPi), at least one environmental sensor (CEj), and a programmable device, said programmable device being configured to: receive, during all or part of a time period preceding the evaluation time (t e), at least one first signal (Spi(t)) acquired by F at least one sensor (CPi), said first signal (Spi(t)) being representative of a first time sequence of behavioral and / or physiological attributes of the person (P) during said time period; receive, during all or part of said time period, at least one second signal (Sej(t)) acquired by F at least one environmental sensor (CEj), said second signal (Sej(t)) being representative of a second time sequence of a state of the environment (E) during said time period;determining, on the basis of the at least one first signal (Spi(t)), a third temporal sequence of emotional signatures of the person (P) during said time period, said third sequence forming a first set of emotional signatures of the person (P), each of the emotional signatures of the person (P) of the third sequence being represented by a position in a space representative of the emotions (R); determining, on the basis of the at least one second current signal (Sej(t)), a fourth temporal sequence of signatures of the environment during said time period, said fourth sequence forming a second set of signatures of the environment (E), each of the signatures of the environment (E) being represented by a position in said space representative of the emotions (R); determine a degree of similarity between the first set of emotional signatures of the person (P) and the second set of signatures of the environment (E), determine the emotional state of the person (P) at the time of evaluation (t e ) from the determined degree of similarity.

2. Determination system (1) according to claim 1, characterized in that the at least one second signal (Sej(t)) is acquired by the at least one environmental sensor (CEj) chosen from: a camera, a webcam, a three-dimensional camera, a microphone, a means of accessing a remote server storing information relating to one or more audiovisual, visual, or audio contents broadcast during all or part of the time period, an electromagnetic detector, and said programmable device being configured to extract from the at least second signal (Ssj(t)) the state of the environment (E), so as to obtain values ​​representative of a state chosen from: a quantity of movement in an audiovisual or visual content, broadcast during all or part of said time period, a sound level and / or a frequency spectrum of an audiovisual or audio content broadcast during all or part of said time period,a sound level and / or a sound frequency spectrum in a close vicinity of the subject, an electromagnetic signal representative of a variation in an electromagnetic signature of a close vicinity of the subject., 3. Determination system (1) according to claim 1 or 2, characterized in that said programmable device is configured to determine the degree of similarity between the first set of emotional signatures of the person (P) and the second set of signatures of the environment (E) via the calculation of a distance in said space representative of emotions (R) between the first set of emotional signatures of the person (P) and the second set of signatures of the environment (E).

4. Determination system (1) according to claim 3, characterized in that said distance is calculated by means of a cosine similarity between a first vector representative of the first set of emotional signatures of the person (P) and the second set of signatures of the environment (E).

5. Determination system (1) according to claim 3 or 4, characterized in that the calculation of said distance comprises a weighting depending on the positions of the emotional signatures of the person (P) of the first set and of the signatures of the environment (E) of the second set in said space representative of the emotions (R).

6. Determination system (1) according to one of claims 3 to 5, characterized in that the calculation of said distance comprises a weighting dependent on time instants corresponding to the emotional signatures of the person (P) in the third time sequence and to the signatures of the environment (E) in the fourth time sequence.

7. Determination system (1) according to one of claims 5 to 6, characterized in that the calculation of said distance is carried out between the emotional signatures of the person (P) of a first sub-sequence of the third temporal sequence and the signatures of the environment (E) of a second sub-sequence of the fourth temporal sequence, said first sub-sequence and second sub-sequence corresponding to a temporal sub-sequence of said temporal period.

8. Determination system (1) according to one of the preceding claims, characterized in that: - the at least one first signal (Spi(t)) is acquired by the at least one sensor (CPi) chosen from: a camera, a webcam, a three-dimensional camera, a microphone, an inertial unit, a physiological sensor, a headset dedicated to electroencephalographic measurements, and in that said programmable device is configured to extract from the at least one first signal (Spi(t)) the behavioral and / or physiological attributes in the first time sequence so as to obtain values ​​representative of at least one of the following attributes from: a gesture of the person (P), a movement of the person (P), an expression of the face of the person (P), a eye movement of the person (P), physiological parameters of the person (P), interaction with an electronic device.

9. Method (100) implemented by computer for determining, at an evaluation time (t e), an emotional state of a person (P) immersed in an environment (E) for the purpose of monitoring an emotional state of said person (P), said method comprising the following steps: receiving, during all or part of a time period preceding the evaluation time (te), at least a first signal (Spi(t)) acquired by F at least one sensor (CPi), said first signal (Spi(t)) being representative of a first time sequence of behavioral and / or physiological attributes of the person (P) during said time period; receiving, during all or part of said time period, at least a second signal (Sej(t)) acquired by F at least one environment sensor (CEj), said second signal (Se t)) being representative of a second time sequence of a state of the environment (E) during said time period;determining, on the basis of the at least one first signal (Sp / t)), a third temporal sequence of emotional signatures of the person (P) during said time period, said third sequence forming a first set of emotional signatures of the person (P), each of the emotional signatures of the person (P) of the third sequence being represented by a position in a space representative of the emotions (R); determining, on the basis of the at least one second current signal (Se t)), a fourth temporal sequence of signatures of the environment during said time period, said fourth sequence forming a second set of signatures of the environment (E), each of the signatures of the environment (E) being represented by a position in said space representative of the emotions (R);determining a degree of similarity between the first set of emotional signatures of the person (P) and the second set of signatures of the environment (E); determine the emotional state of the person (P) at the time of assessment (t e ) from the determined degree of similarity.

10. Method (100) according to claim 9, characterized in that determining the degree of similarity between the first set of emotional signatures of the person (P) and the second set of signatures of the environment (E) comprises calculating a distance in said space representative of emotions (R) between the first set of emotional signatures of the person (P) and the second set of signatures of the environment (E).

11. Method (100) according to claim 10, characterized in that said distance is calculated by means of a cosine similarity between a first vector representative of the first set of emotional signatures of the person (P) and the second set of signatures of the environment (E).

12. Method (100) according to claim 10 or 11, characterized in that the calculation of said distance comprises a weighting depending on the positions of the emotional signatures of the person (P) of the first set and of the signatures of the environment (E) of the second set in said space representative of the emotions (R).

13. Method (100) according to one of claims 10 to 12, characterized in that the calculation of said distance comprises a weighting dependent on time instants corresponding to the emotional signatures of the person (P) in the third time sequence and to the signatures of the environment (E) in the fourth time sequence.

14. Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to any one of claims 9 to 13.

Images