Predicting negative emotional situations in a connected home to promote services that evoke positive emotions

The method addresses the challenge of predicting negative emotional states in connected habitats by analyzing multimodal data from various sources, enabling accurate and non-intrusive emotion prediction and personalized service recommendations.

FR3111044B1Active Publication Date: 2024-11-15ORANGE SA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
FR2020005714
Authority / Receiving Office
FR · FR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-05-29
Publication Date
2024-11-15
Estimated Expiration
2040-05-29

AI Technical Summary

Technical Problem

Existing technologies struggle to robustly predict negative emotional situations in connected habitats, relying on limited data sources like facial expressions, voice, or bodily gestures, and often require intrusive sensors, leading to low acceptance and inaccurate emotion recognition.

Method used

A method utilizing a processing circuit to analyze multimodal data from various sources including individual behavior, home environment, and external context to predict emotional states, using temporal and multimodal correlations to infer an emotion score and trigger appropriate digital services.

Benefits of technology

Enables accurate and non-intrusive prediction of negative emotional states, allowing for personalized services that promote positive emotions by leveraging diverse data sources and optimizing resource use.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000022_0000
    Figure 00000022_0000
  • Figure 00000022_0001
    Figure 00000022_0001
  • Figure 00000022_0002
    Figure 00000022_0002
Patent Text Reader

Abstract

A method is proposed for predicting a user's emotional state and suggesting a predefined digital service. The method includes obtaining signals of interest that respectively measure user behavior, an environmental state, and an external context. The method comprises determining (P5) time series of interest HUM_DB, IN_DB, and OUT_DB; obtaining (P6) a MOD model structuring temporal and multimodal correlations between the values ​​of the time series of interest and the emotional state over time; inferring (P7), at a current time, an emotion score (E) corresponding to a prediction of the emotional state at a later time; triggering (P10) a routine to animate a human-computer interface to suggest a predefined digital service; and, upon obtaining (P11) user feedback, triggering (P12) the predefined digital service.A corresponding computer program, recording medium, and processing circuit is also proposed. See Figure 2 for an abstract.
Need to check novelty before this filing date? Find Prior Art

Description

Title of the invention: Prediction of negative emotional situations in a connected home to promote services that evoke positive emotions. Technical field

[0001] This disclosure falls within the domain of data science.

[0002] More specifically, this disclosure relates to methods for predicting emotional state and triggering service, as well as to computer programs, computer-readable storage devices and processing circuits for implementing such methods. Previous technique

[0003] In the context of home support for vulnerable individuals (e.g., seniors, people with disabilities, isolated individuals, etc.), it is essential to be able to assess the occupants' level of independence. Currently, remote monitoring services implement this type of assessment using actigraphy systems.

[0004] In this context, it appears necessary to also have a better understanding of the occupants' emotions in order to provide a more robust measure of autonomy and appropriate services. This measure is a key indicator for assessing the physical, social, and mental well-being of individuals by medical professionals during remote monitoring services.

[0005] Indeed, emotions can reveal variations in autonomy. It is obvious that boredom, depression, anger, or conversely joy, dynamism, or enthusiasm are indicative of a person's well-being, and these characteristics influence their health and ability to remain at home.

[0006] Analyzing the emotions of people at home is a current area of ​​research.

[0007] Several proposed solutions are described below. These solutions differ in the nature of the data analyzed, including facial expressions, voice, gestures, data from home automation sensors, or data from health sensors. Depending on the nature of the data analyzed, the underlying mathematical models also differ.

[0008] It was proposed in B. Nagarajan and VRM Oruganti, "Group Emotion Recognition in Adverse Face Detection," 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), to implement group emotion recognition from facial images. The technical difficulties encountered for this emotion recognition are related to various heterogeneities in the images to be analyzed. Indeed, in some images, the head People may be partially obscured. Body poses may differ from one image to another, or even from one person to another within the same image. Various indoor and outdoor settings, such as the brightness and orientation of light sources, as well as the resolution of the images being analyzed, can also vary. The proposed algorithm's architecture relies on convolutional neural networks to extract facial features and on a Bayesian network to model the temporal correlations between these features. However, recognizing a person's emotions by focusing solely on facial images is not realistic for a consumer application. Indeed, even without considering the aforementioned technical difficulties, an additional limitation is the very low rate of home camera adoption.

[0009] A neural network model that uses acoustic information, textual information, and three indicators—a temporal indicator, a descriptive indicator, and a geosocial indicator—was proposed in J. Jia et al., "Inferring Emotions From Large-Scale Internet Voice Data," in IEEE Transactions on Multimedia, vol. 21, no. 7, pp. 1853–1866, July 2019. The aim is to provide contextual information that gives meaning to acoustic information received by a voice assistant, based on the duration of a request, the subject of the request, and the user's location. One limitation of this approach is that emotion determination requires both that the person has a voice assistant and that they use it regularly. To more robustly estimate a person's emotional state, it would be desirable to use more sources of information.Another limitation of this approach is the nature of the information extracted from the acoustic data, which is limited to the duration and purpose of the queries. To more robustly estimate a person's emotional state, it would be desirable to be able to extract and use other semantic data from the acoustic data.

[0010] A statistical framework for modeling the dynamics of body gestures in dyadic interactions was proposed in Z. Yang and SS Narayanan, "Modeling Dynamics of Expressive Body Gestures in Dyadic Interactions," in IEEE Transactions on Affective Computing, vol. 8, no. 3, pp. 369-381, 1 July-Sept. 2017. Body gestures are an important channel of nonverbal expression in affective communication. They convey attitudes and emotions that unfold dynamically during interpersonal interaction. First, a generic model based on Gaussian mixture models represents subject-independent gestural variability. Then, each gesture sequence is described as a concatenation of semantic gestures derived from a structure of a parallel-constructed hidden Markov model. A drawback of this approach is that some emotions are not perceptible through body movement, for example, disgust. Furthermore, certain gestures can be common to several types, or classes, of emotions, leading to confusion and incorrect identification. For example, a person's gestures may be similar when they are enthusiastic or happy.

[0011] In D. Fedotov, Y. Matsuda, and W. Minker, "From Smart to Personal Environment: Integrating Emotion Recognition into Smart Houses," 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), the combination of a dialogue system and emotion recognition in a connected home was proposed. The authors believe that these two systems are complementary and that they allow for continuous monitoring of the user's state, real-time interaction, and response to mood changes. The aim is to adapt the dialogue system to the perceived mood, which in this work is conflated with the person's emotion. However, in a home care service for the well-being and health assessment of occupants, a person's mood and emotion must be distinguished.

[0012] A system for improving healthcare services by taking the emotional factor into account was proposed in K. Lin, F. Xia, W. Wang, D. Tian, ​​and J. Song, "System Design for Big Data Application in Emotion-Aware Healthcare," in IEEE Access, vol. 4, pp. 6901-6909, 201. Existing healthcare systems focus solely on logical reasoning and ignore the user's emotional factor, which is nevertheless considered an important factor impacting human health. The sensors used are bracelets, watches, smart glasses, etc. In this approach, the sensors are worn by users, which hinders medical monitoring. Indeed, the acceptance of wearable, intrusive sensors is generally low. Moreover, their use is unsuitable, for example, during periods of care, personal hygiene, etc.

[0013] There is therefore a need to be able to address the aforementioned drawbacks and offer a robust prediction of so-called negative emotional states of people in a connected home, with a view to encouraging these people to experience so-called more positive emotions. Furthermore, it is desirable that the data sources used be multimodal. Summary

[0014] This disclosure improves the situation.

[0015] A method is proposed for predicting a user's emotional state and suggesting a predefined digital service to the user, the method being implemented by a processing circuit connected to a plurality of devices. communicating devices installed in an environment occupied by the user, the process comprising:

[0016] - obtaining a plurality of signals of interest acquired by the equipment communicators, the plurality of signals of interest including at least one signal of interest of a first type measuring user behavior, one signal of interest of a second type measuring a state of the environment and one signal of interest of a third type indicative of a context external to the environment,

[0017] - for each signal of interest obtained, a determination, on the basis of said signal obtained from a time series of interest consisting of a sequence of values ​​as a function of time,

[0018] - obtaining a structuring model of temporal and multimodal correlations between the values ​​of the time series of interest and the user's emotional state over time,

[0019] - based on the model obtained and the time series of interest obtained, a inference, at a current moment, of an emotion score corresponding to a prediction of the user's emotional state at a later moment,

[0020] - based on the inferred emotion score, triggering an animation routine of a human-machine interface to suggest to the user at least one predefined digital service associated with the inferred emotion score, and

[0021] - upon obtaining feedback from the user, received via the human-machine interface, a triggering the predefined digital service via a control signal transmitted to at least one piece of equipment from the plurality of communicating equipment.

[0022] The proposed method aims to address the aforementioned drawbacks and to provide a robust prediction of "negative" emotional situations in a connected home in order to promote services that encourage people to experience "more positive" emotions. An example of emotion classification is described in Parrott, W. (2001), Emotions in Social Psychology, Key Readings in Social Psychology, Philadelphia: Psychology Press. In this example, fifteen classes of emotions are defined: neutral (1), disgusted (2), panicked (3), anxious (4), angry (5), cold anger (6), hopeless (7), sad (8), enthusiastic (9), pleased (10), interested (11), bored (12), ashamed (13), proud (14), and contemptuous (15).In an example implementation using the emotion classes defined above, the emotion classes disgusted, panicked, anxious, angry, desperate, sad, bored and ashamed can be considered negative and the emotion classes enthusiastic, pleased and interested can be considered positive.

[0023] The proposed process is innovative in that the sources used are multimodal, coming from three domains, namely: - information specific to individual behavior, for example related to facial expressions, voice, gestures, etc. - information from the connected home, for example related to temperature, humidity, energy consumption, etc., and - information from outside the home, for example related to weather forecasts, news, etc.

[0024] The proposed process is, moreover, original in its end-to-end design. Indeed, the implementation of this process allows for the processing of raw data received from communicating devices, the inference of an emotion score indicating a predicted emotional state, and the promotion of a tailored service to influence this predicted emotional state. The recognition of "negative" emotions serves as a situational reference for promoting services that evoke "positive" emotions.

[0025] The features described in the following paragraphs may optionally be implemented. They may be implemented independently of each other or in combination with each other:

[0026] In one example, the method further includes collecting signals over time from the plurality of communicating equipment and selecting signals of interest from among the collected signals.

[0027] Indeed, the process is applicable in many possible configurations of domestic installations, in particular no restrictions are imposed on the nature of the communicating equipment present in the domestic installation and the sensors equipping them.

[0028] The relative relevance of the collected signals for predicting the emotional state of residents can, for example, be evaluated during a model calibration phase, so that it is possible, based on this evaluation, to select only those signals obtained that have a proven correlation with a subsequent emotional state of the user. Thus, the use of processing circuit resources is optimized for predicting the user's emotional state.

[0029] In one example, the method further includes a time segmentation of the signals of interest, and the time series of interest are determined on the basis of the segmented signals.

[0030] Time segmentation makes it possible to obtain time series of interest that are chronologically synchronized. For example, a time segmentation into 30-second intervals results in all time series of interest having, for each 30-second interval, an associated value of a quantity of interest. This time synchronization makes it possible, for example, to establish multinodal correlations between the values ​​of different quantities. of interest during the same time interval, or during consecutive time intervals. Thus, it is possible to structure in the model, for example, a correlation between the user's speech rate during a given time interval and their gestures during a subsequent time interval.

[0031] In one example, the method further includes processing the signals of interest, such as low-pass filtering, normalization, or resampling, and the time series of interest are determined on the basis of the processed signals.

[0032] This processing generally allows for the shaping of signals of interest in order to infer the user's emotional state. For example, low-pass filtering reduces noise, normalization standardizes the data, and data resampling synchronizes the sources.

[0033] In one example, the values ​​of the time series of interest are obtained by classifying the signals of interest by modality according to a discrete scale of values ​​associated with said signal.

[0034] This classification by modality, by synthesizing the information contained in the signals of interest, makes it possible to highlight recurring sub-sequences within each time series of interest. Each recurring sub-sequence thus highlighted can serve as the basis for a temporal correlation. Thus, classifying a signal of interest by modality facilitates the detection of temporal correlations with respect to that signal of interest.

[0035] Similarly, classifying several signals of interest by modality has the effect of promoting the detection of multimodal correlations between these signals of interest.

[0036] In one example, the temporal and multimodal correlations of the model are obtained by learning on the basis of:

[0037] - for each time series of interest, of a reference subsection formed of a portion of the sequence of values, the portion covering a reference time period, and

[0038] - of a knowledge base listing, for a plurality of moments in the period temporal reference, a reference emotion score indicative of the user's emotional state at that instant.

[0039] Thus, after this learning, the time series of interest continue to be acquired in the same way as during the reference time period, using the same communicating equipment.

[0040] A subsequent emotion score of the user can then be inferred based on the correlations established, under the same conditions, between the emotion scores of the knowledge base and the reference subsections of the time series of interest.

[0041] In one example, the model includes a given temporal correlation between several values ​​of a given time series of interest, and the inference of the emotion score is based on the given temporal correlation.

[0042] A particular subsequence can be identified within a time series of interest. For example, a facial expression indicating boredom can be detected repeatedly over several consecutive moments. It is thus possible to infer that at a later moment, this facial expression will persist. On this basis, it is therefore possible to infer a subsequent emotional state of "boredom".

[0043] In one example, the time series of interest are grouped into a plurality of informational domains, the model includes, for at least one given informational domain, a given multimodal correlation between a plurality of time series of interest of said given informational domain, and the inference of the emotion score is based on the given multimodal correlation.

[0044] For example, an emotion score can be inferred based on a multimodal correlation with news and weather information. For example, sunny weather, associated with a news event perceived positively by a given person, can generally be correlated with a positive emotional state of that person.

[0045] In one example, the method includes obtaining a latent time series formed from a sequence of values ​​as a function of time, the model obtained includes a given temporal correlation between several values ​​of the latent time series, the inference of the emotion score includes a determination, at the current time, of the value of the latent time series at the current time based on the given temporal correlation, and the inference of the emotion score is based on the determined value of the latent time series.

[0046] A latent time series allows for the representation of a non-measurable quantity, for example, in the absence of a suitable sensor, but which is nevertheless correlated with both one or more measurable quantities and the person's emotional state. This latent time series can be manipulated as such by the model. Thus, it is possible to structure, using the model, a temporal correlation between different values ​​of any time series, whether it relates to a measurable quantity or a latent quantity.

[0047] In one example, the method includes, based on the model and the time series obtained, a determination, at a current time, of a confidence level associated with the inferred emotion score, the method further includes a comparison of the determined confidence level with a threshold, and the triggering of the routine is based on the result of the comparison.

[0048] Thus, for a given inferred emotion score, it is possible to recommend a predefined service to a user, for example, only if multimodal and temporal correlations have allowed the emotion score to be inferred with a sufficient level of confidence. This avoids suggesting services to a user based on erroneous predictions of their subsequent emotional state.

[0049] In one example, a plurality of suggested digital services are defined by respective links stored in a lookup table in association with respective emotion score thresholds and, if a threshold is exceeded by the user's emotion score, at least one link of a corresponding digital service is read to suggest said corresponding digital service to the user via the human-machine interface.

[0050] Thus, a service is only recommended to the user if the intensity of a predicted emotion is sufficiently high. This has the effect of reserving service suggestions for the user at times when the execution of these services would be most useful to them.

[0051] In one example, the respective links are stored in the lookup table according to a preference order; if a threshold is exceeded by the user's emotion score, a plurality of corresponding digital service links are read according to the preference order to suggest said corresponding digital services to the user via the human-machine interface; the user's response to the issued invitation includes a selection of a digital service from among said corresponding digital services, and the preference order is updated on the basis of said selection.

[0052] Thus, a selection of services is offered to the user and the services usually preferred from this selection will be put more, or more frequently, in the foreground.

[0053] A computer program is also proposed comprising instructions for implementing the above process when this program is executed by a processor.

[0054] A non-transient recording medium readable by a computer is also proposed on which a program is recorded for the implementation of the above process when this program is executed by a processor.

[0055] A processing circuit is also proposed comprising a processor connected to the above non-transient recording medium and to a communication interface. Brief description of the drawings

[0056] Other features, details and advantages will become apparent from the detailed description below and from the analysis of the accompanying drawings, in which: Fig. 1

[0057] [Fig.1] schematically illustrates the structure of a processing circuit, in a particular embodiment, for the implementation of the proposed determination process. Fig. 2

[0058] [Fig.2] represents a flowchart of a general algorithm of a computer program, in a particular example of embodiment, for the implementation of the proposed determination process. Fig. 3

[0059] [Fig.3] represents, in the form of a dynamic Bayesian network, an emotion prediction model that can be derived, in a particular example of realization, from the proposed determination process. Description of the implementation methods

[0060] The drawings and description below contain, essentially, elements of a definite nature. They may therefore not only serve to better explain this disclosure, but also contribute to its definition, if necessary.

[0061] Reference is now made to [Fig.1], which represents an example of a processing circuit integrated into a communicating device, installed in a space occupied by a person, such as a dwelling.

[0062] The processing circuit includes at least one PROC 100 processor connected to at least one MEM 200 memory and to at least one COM 300 communication interface.

[0063] A communicating device is characterized by its ability to receive and transmit information via the communication interface. For example, in a home installation, different communicating devices can be configured to communicate with each other through a local communication network and / or a wide area network. Communication can be wired and / or wireless.

[0064] Some communicating equipment may also be equipped with sensors enabling the collection of data.

[0065] Examples of communicating equipment include smartphones, tablets, computers, connected televisions, network gateways, connected speakers, connected watches, connected bracelets or headbands, household appliances, weather stations, electricity, gas or water meters, connected lighting, connected cameras, connected thermostats, etc.

[0066] Reference is now made to [Fig.2], which represents a flowchart of a general algorithm of a computer program stored on the MEM memory (200) of the processing circuit shown in [Fig.1].

[0067] This processing circuit is configured to continuously obtain, or receive, a stream of signals via SIGN OBT (PI). Each signal carries at least one type of digital information.

[0068] If the processing circuit is integrated into a communicating equipment equipped with one or more sensors, then the reception of signals can be carried out by acquisition using the said sensor(s).

[0069] For example, if the communicating equipment includes a camera, then it is possible to collect a video recording of the person, as a signal carrying several types of digital information. This video recording can be analyzed by the processing circuit in order to identify this digital information. It is thus possible to collect different facial features or expressions, different gestures or postures, the person's movements, ambient lighting, etc.

[0070] Signal reception can be performed in parallel via the communication interface of the processing circuit, receiver. Thus, it is possible to receive signals emitted by other communicating equipment, transmitters, in the home installation.

[0071] For example, a smart speaker, as a transmitting communicating device, can collect an audio recording as a signal carrying digital information. Processing this audio recording directly by the smart speaker can extract, for example, an ambient sound level, a person's speech rate, an estimated position of the person, etc. It is possible to transmit each type of extracted digital information separately. In other words, the smart speaker can transmit a plurality of signals. Each signal carries a type of digital information to the receiving communicating device.

[0072] Alternatively, the connected speaker can directly transmit the audio recording to the receiving communicating equipment, without prior processing. In this case, the connected speaker transmits an audio signal carrying several types of digital information.

[0073] For example, a weather station, as a communicating transmitting device, can be equipped with sensors that continuously collect various digital home automation data. Examples include ambient temperature in different rooms, humidity, atmospheric pressure, carbon dioxide levels, etc. This digital home automation data can be transmitted in the form of time series, therefore in the form of sequences of time-stamped values.

[0074] Generally speaking, depending on the nature and configuration of the various pre-existing communicating devices within a particular domestic installation, it is possible to obtain a wide variety of digital information. In principle, the digital information thus obtained is heterogeneous in both its nature and its format.

[0075] Three categories of information obtained are defined below.

[0076] A first category of information concerns information describing a person's behavior. For example, this could include facial features, voice, gestures, etc. This type of information can come, for example, from communicating equipment including audio and / or video sensors. Subcategories can be defined. For example, a first subcategory could encompass facial features. A second subcategory could group voice features.

[0077] A second category of information concerns information related to the space occupied by the person. For example, this could include door opening indicators, water, gas, or electricity consumption readings, temperature, light, or noise level measurements, etc. This type of information can be acquired, for example, by suitable sensors present on various connected home devices.

[0078] A third category of information concerns one or more pieces of information related to a context outside the home. For example, this could include weather information, news feeds, air pollution indicators, etc. This type of information can be downloaded by a communicating device, for example from a remote server.

[0079] In the context of disclosure, it is considered that at least one digital piece of information is obtained for each category of information defined above.

[0080] Other digital information not belonging to any of the aforementioned categories, or belonging to several of the aforementioned categories, may be obtained.

[0081] In parallel with the collection of data by receiving the signal stream, it is possible for the processing circuit to operate, for each category or subcategory of information, a SIGN SELEC (P2) selection of signals of interest to be processed.

[0082] For example, it is possible to process only a selection of 50 home automation pieces of information from 200 collected home automation pieces of information.

[0083] Similarly, it is possible to process only a selection of 10 facial features from among 50 collected.

[0084] The advantage of making such a selection is to minimize the exchange of data between communicating equipment by ignoring characteristics that have no influence on the emotional state of the person.

[0085] For example, it is possible to monitor a person's emotional state over an extended period of several weeks or months. This monitoring can be implemented through repeated iterations of the proposed method. This monitoring may reveal, through cross-referencing, that certain information obtained is unrelated to the person's emotional state. Based on this finding, this information can be disregarded in the future.

[0086] Furthermore, a sociological study may highlight that some information obtained may be ignored a priori.

[0087] In order to homogenize the signals for their subsequent analysis, it is possible for the processing circuit to define SIGN WIND (P3) an analysis window, or a measurement step.

[0088] For example, home automation information such as an ambient sound level can be recorded continuously, segmented into consecutive intervals of a predefined duration, and represented by a time series corresponding to a succession of average ambient sound levels during each elapsed interval.

[0089] For example, meteorological information such as an outside ambient temperature can be recorded in a discrete manner so as to form a time series corresponding to a succession of instantaneous values.

[0090] Thus, it is possible to represent each type of information obtained as a time series formed from a sequence of time-stamped values. In other words, it is possible to associate with each measurement a corresponding instant i and with each time series comprising n measurements a sequence of corresponding consecutive instants (0, 1, ..., i-1, i, i+1, ... n-1, n), each instant corresponding to a predefined time interval.

[0091] In an example implementation, it is possible to segment the data every minute without overlap.

[0092] The processing circuit can implement various specific SIGN PROC (P4) processing of the collected and segmented signals of interest.

[0093] The processing includes, but is not limited to, low-pass filtering to denoise the information, normalization to standardize the data, data resampling to synchronize the sources, etc.

[0094] The signals thus obtained can be classified SIGN CLASS (P5) by modality. The term modality designates the different values ​​that can be taken with respect to a type of information.

[0095] In one example implementation, the facial features of individuals can be classified on an emotional intensity scale from 1 to 5, to respectively designate the intensity of emotions such as neutral, joyful, disgusted, angry, or bored. For voices, speech rate can, for example, be classified on a scale of 1 to 4, designating a slow, neutral, fast, or very fast rate, respectively. The intensity of gestures can be classified on a scale of 1 to 3, with 1 designating the person at rest, 2 designating gestures of normal intensity, and 3 designating energetic gestures.

[0096] Similarly, information characterizing the house can be categorized on a scale of values. For example, a switch can be used to determine at any given moment whether a door is open or closed. The time series resulting from this classification by category could be, for example, an alternation of values ​​equal to 1 when the door is open and 2 when the door is closed. Water, gas, or electricity consumption can be compared to predefined thresholds and classified by category according to a scale, for example, from 1 to 4, corresponding respectively to very low, low, high, or very high consumption. The same applies to ambient temperature, light, and noise.

[0097] Contextual information from outside the home can also be classified. For example, a scale from 1 to 4 could designate sunny, rainy, snowy, or cloudy weather, respectively. Major news items of the day can also be reported using such scales. Thus, it is possible to assign a rating of 1 if the main news item is political, 2 if it is sports news, and so on. It is also possible to consider each type of news item separately and assign a rating to each type (political, sports, health, leisure, celebrity, etc.) based on the presence or absence of that type of news, an assessment of its relevance to the individual, and / or its reassuring or worrying nature.

[0098] Thus, the databases HUM_DB, IN_DB, OUT_DB are constituted to store respectively the history of pre-classified information for the three categories of information, or informational domains, describing the state of the inhabitants (e.g. faces, voices, gestures, etc.), of the interior of the dwelling (e.g. temperature, humidity, energy consumption, etc.) and of the exterior of the dwelling (e.g. news, weather, pollution index, etc.).

[0099] A pre-learned MOD model is loaded MOD LD (P6) by the processing circuit. The MOD model structures temporal and multimodal correlations between the different information sources.

[0100] Reference is now made to an example implementation in which the MOD model is a dynamic Bayesian network. The inventors have found, by validation experimental, the robustness of predicting a person's emotional state based on such a model.

[0101] Of course, alternatively, any type of probabilistic model establishing temporal and multimodal correlations is applicable. However, the results thus obtained do not necessarily have equivalent robustness.

[0102] The MOD model can be predetermined, or learned, during a calibration phase.

[0103] This determination is based on numerical information previously collected over a predefined period, referred to as the reference period.

[0104] For example, it is possible to rely on:

[0105] - a subsection of the databases HUM_DB, IN_DB, OUT_DB, corresponding to the data collected over a period of one week, and

[0106] - an EM_DB knowledge base labeling the person's emotions during during the same period.

[0107] In an example implementation, fifteen classes of emotions can be used, namely neutral (1), disgusted (2), panicked (3), anxious (4), angry (5), cold anger (6), desperate (7), sad (8), enthusiastic (9), pleased (10), interested (11), bored (12), ashamed (13), proud (14) and contemptuous (15).

[0108] The labels, designating the classes of emotions, can be provided by the person themselves, for example in the form of a logbook. Alternatively, the labels can be provided by a trusted third party, such as family, a close friend, medical professionals, etc.

[0109] Emotion labels thus allow learning of temporal correlations and multimodal correlations between the databases HUM_DB, IN_DB, OUT_DB and the emotion labels EM_DB.

[0110] These correlations constitute an example of a customized MOD model provided in an initial operating version. The MOD model is, of course, subject to change over time depending on the evolution of the accumulated numerical data.

[0111] Different MOD models, corresponding to different people, in different dwellings, with different home installations, can be combined to create generic, standard models that can be applied by default to a new person in a new dwelling with a home installation comprising a given set of communicating devices. In this way, it is possible to eliminate, in certain implementation examples, the need for a calibration phase.

[0112] Reference is now made to [Fig.3], representing, in an example of implementation, the MOD model in the form of a dynamic Bayesian network.

[0113] A person's emotional state E at a given time i potentially depends on the person's recent emotional history. In other words, there is a temporal correlation between the person's emotional state E at a future time i+1 and at the current time i, the previous time i-1, and so on. This temporal correlation is represented in [Fig. 3] by the arrows connecting the emotional states Em, E, and Ei+i.

[0114] Furthermore, there are multimodal correlations between the emotional state Ei+i of the person at a future time i+1, on the one hand, and different variables at the current time i, on the other. These multimodal correlations are represented in [Fig. 3] by the arrows connecting L, F, Vi, Gi, T, H, Ci, Ni, and Mi with Ei+i.

[0115] We can cite in particular variables of the first informational domain, designating states of the person, such as characteristics of the face, noted F, of the voice, noted V or of gestures, noted G.

[0116] We can also mention variables from the second informational domain, designating home automation information, such as indoor temperature, denoted T, indoor humidity, denoted H, or energy consumption, denoted C.

[0117] We can also mention variables from the third informational domain, designating contextual information external to the home, such as news information, denoted N, or weather, denoted M.

[0118] Latent (unobservable) variables, denoted L, can also be mentioned. Such variables represent unmeasurable quantities, for example, due to the absence of sensors that would allow them to be measured. In this case, a quantity L is modeled by multimodal and / or temporal correlation with other obtained data. For example, indoor brightness can be correlated with natural lighting, i.e., sunlight, and with artificial lighting. Thus, brightness can be taken into account by the MOD model even in the absence of a brightness sensor. Indeed, assuming the correlations mentioned above, brightness at a given time can be modeled using meteorological information, operating data from smart lighting systems, and / or modeled brightness values ​​from previous times.

[0119] It should also be noted that the information produced by the latent variables L, the inhabitants, the home and the external context are time-dependent, that is to say that the times i-1 and i, in particular, influence the time i+1. In other words, there are internal temporal correlations within each of these variables.

[0120] It should also be noted that voice and gesture influence facial features. These multimodal correlations are represented in [Fig. 3] by the arrows connecting V; and G; with Fi+i. Similarly, the face and voice influence gestures and the face and gestures influence the voice.

[0121] Similarly, information from the house is correlated (for example: temperature and humidity influence energy consumption, etc.). Also, information from the external context can be linked (e.g., weather and news, etc.).

[0122] In an example implementation, it is possible to exclude a priori certain correlations.

[0123] For example, it is possible to predict that, in the MOD model, no correlation is structured between different informational domains. No arrow connects variables F, V, and G with variables T, H, and C in [Fig. 3], for instance. These informational domains are then considered independent. For example, it is possible to consider that the information produced by the inhabitants is not directly influenced by either home automation information or external contextual information, and vice versa.

[0124] Similarly, it is possible to predict that a latent variable will be determined only by temporal correlation and not by multimodal correlation. Thus, this latent variable is independent of the measured information.

[0125] The correlations that can be excluded a priori concern only the interactions between different obtained and / or latent variables, and not the correlations between a obtained or latent variable, on the one hand, and an emotional state, on the other. Indeed, the emotional states E are potentially correlated with the obtained variables for each informational domain and with each latent variable L.

[0126] The processing circuit infers, or predicts EMOT PRED (P7), an emotional state E of a person at a later time i+1 based on temporal and multimodal correlations structured by the MOD model.

[0127] These correlations are applied to the recent history of the information obtained, that is to say to the information associated with the current time i and to the information associated with one or more previous times i-1, i-2, etc.

[0128] The information obtained includes: - those from the HUM_DB, IN_DB, OUT_DB, and databases - the values ​​of the latent variables L used by the MOD model.

[0129] Said information obtained also includes the person's current emotional state, as inferred from a previous iteration.

[0130] Thus, the proposed method makes it possible, at any moment, both to determine the current emotional state of a person and to predict its evolution over time.

[0131] In some embodiment examples, a confidence measure Z can be associated CONF COMP (P8), by the processing circuit, with the emotional situation of the person at time i+1 inferred by the MOD model.

[0132] For example, a maximum likelihood value is calculated for the emotional state being in state 3 (i.e., the class representing "panicked"). The higher this value, the greater the confidence in the prediction.

[0133] In some embodiment examples, it may be decided to take into account TEST DIFF (P9) both the inferred emotional situation and the confidence measure Z thus determined to offer a service.

[0134] For example, a double condition may be provided, such as:

[0135] - if the confidence measure Z is greater than a predefined threshold A (on a scale of (0 to 1, the threshold can be set to 0.8 in an example implementation), and

[0136] - if the person's predicted emotional state is considered "negative "

[0137] then a suitable service can be offered to the person.

[0138] Otherwise, it may be planned to offer no service, to stop the current iteration of the process at the current time i, and to trigger a subsequent iteration of the process at the later time i+1.

[0139] In an example implementation, the "negative" classes can be chosen as follows: disgusted (2), panicked (3), anxious (4), angry (5), coldly angry (6), hopeless (7), sad (8), bored (12) and ashamed (13).

[0140] The emotional situation at the lower moment being inferred, a service recommendation, adapted to the prediction, can be triggered by the processing circuit.

[0141] Thus, an animation routine for a human-machine interface of any communicating equipment in the home installation is triggered to present SERV PRES (P 10) the recommended service to the person.

[0142] In one example implementation, the service recommendation is executed on the best broadcast channel for the person, such as the living room TV, a smart speaker, a smartphone, etc.

[0143] The most suitable communicating equipment for this purpose can be selected, for example, on the basis of the functionalities offered by the different communicating equipment in the installation.

[0144] Another criterion may be an estimate of the distance, at the current moment, between the different communicating devices and the person, allowing the choice, among the different communicating devices of the installation capable of triggering the proposed services, of the one that is closest to the person at the current moment.

[0145] The animation routine allows the person to be suggested at least one predefined digital service associated with the inferred emotion score.

[0146] For example, if the predicted situation is "panic", the associated services in a PREF preference model might be, in order of preference, "a recommendation of a comedy video", "an incentive to call a friend", or "a "Increased home comfort through automatic adjustment of lighting and heating."

[0147] Other examples of services can be described for each "negative" emotional situation according to people's preferences and habits.

[0148] These preferences can be defined by the individuals themselves or by a trusted third party, such as family, a close friend, or the medical profession.

[0149] These preferences can be defined, reorganized or updated automatically by learning from services used in the past in correlation with predicted emotions.

[0150] The objective is always to evoke or maintain a positive emotional situation through appropriate services.

[0151] The processing circuit can then evaluate TEST AGR (PI 1) the person's response to the request for the appropriate service. Indeed, if a list of possible services is suggested, each service on the list can be either activated, refused, or ignored by the person.

[0152] If a service is activated by the person, then it is possible to execute the proposed SERV EXEC (P12) service.

[0153] It is also possible, where appropriate, to update PREF UPD (P13) the person's PREF preference model according to the predicted emotional state and the service accepted.

[0154] It is also possible to update the MOD model based on the person's feedback to the request for the adapted service. Indeed, it can be provided that the person can report an incorrect prediction of their subsequent emotional state, or can report when the prediction is correct, in order to improve the predictions in subsequent iterations of the process.

[0155] Finally, when a proposed service is activated and executed, or when a proposed service is refused, or when no proposed service is activated after a predefined period of time, the iteration of the process at the current time ends. The processing circuit can then initiate a new iteration of the process for a later time.

Claims

1. Claims Method for predicting an emotional state of a user and triggering a digital service, the method comprising the following steps implemented by a processing circuit connected to a plurality of communicating devices installed in an environment occupied by the user: - obtaining a plurality of signals of interest acquired by the communicating equipment, the plurality of signals of interest comprising at least one signal of interest of a first type measuring a behavior of the user, a signal of interest of a second type measuring a state of the environment and a signal of interest of a third type indicative of a context external to the environment, the signals of interest being collected during a reference period, - for each signal of interest obtained, a determination (P5), on the basis of said signal obtained, of a time series of interest (HUM_DB, IN_DB, OUT_DB) formed of a sequence of values ​​as a function of time, - obtaining (P6) a MOD model structuring temporal and multimodal correlations between the values ​​of the time series of interest and the emotional state of the user as a function of time, - on the basis of the model obtained and the determined time series of interest, an inference (P7), at a current time, of an emotion score (E) corresponding to a prediction of an emotional state of the user at a later time, with a view to updating the temporal and multimodal correlations within the MOD model by machine learning, the method further comprising, after learning, a use of the MOD model to constitute a standard model and the following steps implemented by a processing circuit connected to a new set of communicating equipment installed in a new environment occupied by a new user: a use of the standard model to infer an emotion score (E) corresponding to a prediction of an emotional state of the new user, on the basis of the inferred emotion score (E) corresponding to the prediction of the emotional state of the new user, a triggering (P 10) of an animation routine of an interface human-machine interface to suggest to the new user at least one predefined digital service associated with the inferred emotion score (E) corresponding to the prediction of the emotional state of the new user, and upon obtaining (PI 1) feedback from the new user, received by the human-machine interface, a triggering (P12) of the predefined digital service via a control signal transmitted to at least one device of the new set of communicating devices.

2. Method according to the preceding claim, further comprising a collection (PI) of signals over time from the plurality of communicating devices and a selection (P2) of the signals of interest from among the collected signals.

3. Method according to one of the preceding claims, further comprising a temporal segmentation (P3) of the signals of interest, and in which the time series of interest are determined on the basis of the segmented signals.

4. Method according to one of the preceding claims, further comprising processing (P4) of the signals of interest, such as low-pass filtering, normalization, or resampling, and in which the time series of interest are determined on the basis of the processed signals.

5. Method according to one of the preceding claims, in which the values ​​of the time series of interest are obtained by classifying the signals of interest by modality according to a discrete scale of values ​​associated with said signal.

6. Method according to one of the preceding claims, in which the temporal and multimodal correlations of the MOD model are obtained by learning on the basis of: - for each time series of interest, a reference subsection formed from a portion of the sequence of values, the portion covering a reference time period, and - a knowledge base listing, for a plurality of instants of the reference time period, a reference emotion score indicative of the emotional state of the user at said instant.

7. Method according to one of the preceding claims, in which: - the model comprises a given temporal correlation between several values ​​of a given time series of interest, and - the inference of the emotion score is based on the given temporal correlation.

8. Method according to one of the preceding claims, in which: - the time series of interest are grouped into a plurality of informational domains, - the model comprises, for at least one given informational domain, a given multimodal correlation between a plurality of time series of interest of said given informational domain, and - the inference of the emotion score is based on the given multimodal correlation.

9. Method according to one of the preceding claims, in which: - the method comprises obtaining a latent time series formed from a sequence of values ​​as a function of time, - the obtained model comprises a given temporal correlation between several values ​​of the latent time series, and - the inference (P7) of the emotion score comprises a determination, at the current time, of the value of the latent time series at the current time on the basis of the given temporal correlation, and the inference (P7) of the emotion score is based on the determined value of the latent time series.

10. Computer program comprising instructions for implementing the method according to one of claims 1 to 9 when this program is executed by a processor.

11. Non-transitory recording medium readable by a computer on which is recorded a program for implementing the method according to one of claims 1 to 9 when this program is executed by a processor.

12. Processing circuit comprising a processor PROC (100) connected to a communication interface COM (300) and to a non-transitory recording medium MEM (200) according to claim 11.