Method for determining an emotion of a user present in a vehicle, associated system and computer program

Abstract

Method for determining an emotion of a user present in a vehicle, associated system and computer program This method comprises the following steps implemented by the processing unit: - acquisition (100) of a plurality of images of the user over a duration T1, - estimation (110) of a detection confidence score of each emotion, among the predefined set of emotions, respective to each acquired image, - calculation (120) of a set of stable emotions over said duration T1, from the detection confidence scores of each emotion for said plurality of acquired images, - determination (130), by an artificial intelligence subsystem, of a predominant emotion of the user over a duration T2, from the calculated stable emotions and at least one vehicle data item, the duration T2 being at least equal to the duration T1, and - transmission (140) of the determined predominant emotion to at least one service using the determined predominant emotion.Figure for abstract: Figure 2.

Description

Title of the invention: Method for determining an emotion of a user present in a vehicle, associated system and computer program

[0001] The present invention relates to a method for determining an emotion of a user present in a vehicle.

[0002] The present invention also relates to a system for determining an emotion of a user present in an associated vehicle and an associated computer program.

[0003] The invention then relates to the field of applications embedded in vehicles, in particular motor vehicles, and more particularly applications for increasing the comfort and safety of users.

[0004] In this field, services taking into account the emotions of users, and in particular of the driver of a vehicle, have been developed.

[0005] Although emotion recognition is very complex, more and more applications are trying to automate this type of recognition.

[0006] It is known to analyze an image, such as a photo, in order to detect the emotional state of a person.

[0007] However, in practice, a user's emotions are not fixed as in a photo, but fluctuate greatly over time. This may be due to the type of emotion, the intensity of the emotion felt by the user, or even associated daily or random external factors that may affect the user's emotional state.

[0008] When emotion detection systems are used continuously or when detection is performed from several images, the analysis is often imprecise because several similar emotions can be detected and present strong variations between two successive images. The fluctuation of emotions therefore seems a priori random and unpredictable.

[0009] The aim of the invention is then to propose a method and an associated system making it possible to determine a predominant emotion of a user more precisely.

[0010] For this purpose, the subject of the invention is a method for determining an emotion of a user present in a vehicle, from among a predefined set of emotions, implemented by a determination system, the determination system comprising a camera connected to an on-board computer, the camera being configured to capture images of said user, the on-board computer comprising a processing unit. This method comprises the following steps implemented by the processing unit: a. acquisition of a plurality of images of the user over a duration Tl,

[0011] b- estimation of a detection confidence score of each emotion, among the predefined set of emotions, respective to each acquired image,

[0012] c- calculation of a set of stable emotions over said duration Tl, from the detection confidence scores of each emotion for said plurality of acquired images,

[0013] d- determination, by an artificial intelligence subsystem, of a predominant emotion of the user over a duration T2, from the calculated stable emotions and at least one vehicle data item, the duration T2 being at least equal to the duration Tl, and

[0014] e- transmission of the determined predominant emotion to at least one service using the determined predominant emotion.

[0015] The use of at least one vehicle data and the application of the artificial intelligence subsystem make it possible to carry out an improved method capable of determining the predominant emotion more precisely while maintaining an acceptable duration between the acquisition step and the transmission step.

[0016] According to other advantageous aspects of the invention, the method for determining an emotion of a user present in a vehicle comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations:

[0017] -it comprises an additional step of delivering at least one personalized service for the user using at least one piece of vehicle equipment, depending on the predominant emotion.

[0018] - At least one vehicle data item is chosen from the following information: in traffic formations in a vehicle environment, vehicle geolocation, vehicle internal temperature, vehicle external temperature, time, and vehicle trip history.

[0019] - the step of determining the user's determined predominant emotion on the duration T2, by the artificial intelligence subsystem, is carried out from the calculated stable emotions, from T at least one vehicle data and from at least one user data.

[0020] - At least one user data item is chosen from the following information: electronic calendar, birthday, gender, address, cell phone notifications, and call history.

[0021] - the calculation of the set of stable emotions over said duration Tl, is based on a score detection confidence greater than a predefined threshold on the set of detection confidence scores of each emotion for the plurality of acquired images.

[0022] - the artificial intelligence subsystem implements a pre-a neural network reliably trained by deep learning with a qualified dataset.

[0023] The invention also relates to a system for determining an emotion of a user present in a vehicle, from among a predefined set of emotions, the system comprising a camera connected to an on-board computer, the camera being configured to capture images of said user, the on-board computer comprising a processing unit, the processing unit being configured to implement:

[0024] - a module for acquiring a plurality of images of the user over a duration T1,

[0025] - a module for estimating a confidence score for detecting each emotion, among the predefined set of emotions, respective to each acquired image,

[0026] - a module for calculating a set of stable emotions over said duration T1, from the detection confidence scores of each emotion for said plurality of acquired images,

[0027] - a module for determining, by an artificial intelligence subsystem, a predominant emotion of the user over a duration T2, from the calculated stable emotions and at least one data item provided by the vehicle or by the vehicle and a user of the vehicle, the duration T2 being at least equal to the duration T1;

[0028] - a module for transmitting the predominant emotion determined to at least one service using the determined predominant emotion.

[0029] The invention also relates to a computer program comprising executable code instructions, which, when executed by a programmable system, implement a method for determining an emotion of a user present in a vehicle as defined above.

[0030] The invention will appear more clearly on reading the description which follows, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0031] [Fig-1] [Fig.l] is a representation of an emotion determination system of a user present in a vehicle;

[0032] [Fig.2] [Fig.2] is a flowchart of a method for determining emotions of a user present in a vehicle;

[0033] [Fig.3] [Fig.3] is an example of a table comprising a result of estimating a detection confidence score for each emotion among a predefined set of emotions from an acquired image.

[0034] The expression “substantially equal to” corresponds to a value with an accuracy of ±5%.

[0035] [Fig.l] schematically represents a system 10 for determining the emotions of a user present in a vehicle 12, for example a motor vehicle.

[0036] The system 10 determines an emotion of a user, from a predefined set of emotions. The system 10 is used in a passenger compartment 14 of the vehicle 12 as shown in [Fig.l] by way of example.

[0037] The user 20 present in the vehicle 12 is a passenger or the driver of the vehicle 12.

[0038] In the example of [Fig.l], the system 10 comprises a camera 16 and an on-board computer 18.

[0039] The camera 16 is arranged so as to detect a user 20 present in the passenger compartment 14.

[0040] Alternatively, the system 10 comprises a plurality of cameras 16, each camera being arranged relative to a seat of the passenger compartment 14 so as to capture an image of the user seated in the seat.

[0041] The or each camera 16 is configured to acquire digital images.

[0042] The or each camera 16 is connected to the on-board computer 18, for example to the by means of a wired connection, allowing the acquired digital images to be transmitted to the on-board computer 18.

[0043] The on-board computer 18 forms a programmable system configured to execute a method for determining an emotion of a user according to the invention.

[0044] The on-board computer 18 comprises a processing unit 22, comprising one or more calculation processors and an electronic memory 24. The processing unit 22 comprises or is connected to an artificial intelligence subsystem 25.

[0045] The electronic memory 24 comprises an acquisition module 26 for a plurality of images of the user over a duration T1, an estimation module 28 for a detection confidence score of each emotion, among the predefined set of emotions, respective to each acquired image, a calculation module 30 for a set of stable emotions over the duration T1, from the detection confidence scores of each emotion for the plurality of acquired images, a determination module 32, implementing the artificial intelligence subsystem 25, of a predominant emotion of the user over a duration T2, from the calculated stable emotions and at least one data item of the vehicle, the duration T2 being at least equal to the duration T1, and a transmission module 34 for transmitting the determined predominant emotion to at least one service using the determined predominant emotion.

[0046] The electronic memory 24 further comprises a first recovery module 36 of at least one piece of data from the vehicle 38.

[0047] For example, the first recovery module 36 T at least one piece of data from the vehicle 38 has access to other memories of the on-board computer 18.

[0048] Preferably, at least one piece of data of the vehicle 38 is chosen from the following information: traffic information in an environment of the vehicle, geolocation of the vehicle, internal temperature of the vehicle, external temperature of the vehicle, time, and history of the vehicle's journeys.

[0049] As an optional addition, the electronic memory 24 also comprises a second recovery module 40 of at least one user data item 42.

[0050] Of course, the second recovery module 40 is implemented with the authorization of the user 20.

[0051] Preferably, the at least one piece of data of the user 42 is chosen from the following information: electronic diary, date of birth, gender, address, mobile phone notifications, and call history.

[0052] For example, the second recovery module 40 is connected to a user device, for example a user telephone (or smartphone), so as to recover user data 42.

[0053] In the example of [Fig.l], the acquisition module 26, the estimation module 28, the calculation module 30, the determination module 32, the transmission module 34 and the recovery module 36, as well as the optional addition of the backup module 40, are each produced in the form of software, or a software brick, executable by the processing unit 22, and form software which, when executed by a programmable system, implements a method for determining an emotion of a user present in a vehicle.

[0054] According to a variant, the acquisition module 26, the estimation module 28, the calculation module 30, the determination module 32, the transmission module 34 and the recovery module 36, as well as, as an optional addition, the backup module 40, are each produced in the form of a programmable logic component, such as an FPGA (Field Programmable Gate Array) or an integrated circuit, such as an ASIC (Application Specific Integrated Circuit).

[0055] The artificial intelligence subsystem 25 comprises executable code instructions, for example recorded on an information medium.

[0056] For example, the information medium is a medium readable by the processing unit 22. In another example, the readable information medium is a medium adapted to memorize electronic instructions and capable of being coupled to a bus of a computer system.

[0057] For example, the information medium is a USB key, a floppy disk or floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a ROM memory, a RAM memory, an EPROM memory, an EEPROM memory, a magnetic card or an optical card.

[0058] The artificial intelligence subsystem 25 implements a neural network previously trained by deep learning with a set of qualified data, or training data, to determine a predominant emotion of a user, from a set of stable emotions over the duration T1, the at least one data item of the vehicle 38 and optionally, the at least one data item of the user 42.

[0059] The neural network comprises, for example, an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer.

[0060] For example, each layer comprises neurons taking their inputs from the outputs of the neurons of the previous layer, or from the input variables for the first layer.

[0061] More complex neural network structures can, for example, be envisaged with a layer that can be connected to a layer further away than the immediately preceding layer.

[0062] Each neuron is also associated, for example, with an operation, i.e. a type of processing, to be carried out by said neuron within the corresponding processing layer.

[0063] For example, each layer is connected to the other layers by a plurality of synapses. A synaptic weight is associated with each synapse, and each synapse forms a connection between two neurons. It is often a real number, which takes both positive and negative values. In some cases, the synaptic weight is a complex number.

[0064] As another example, each neuron is capable of performing a weighted sum of the value(s) received from the neurons of the previous layer, each value then being multiplied by the respective synaptic weight of each synapse, or connection, between said neuron and the neurons of the previous layer, then applying an activation function, typically a non-linear function, to said weighted sum, and delivering at the output of said neuron, in particular to the neurons of the following layer which are connected to it, the value resulting from the application of the activation function. The activation function makes it possible to introduce non-linearity into the processing carried out by each neuron. The sigmoid function, the hyperbolic tangent function, the Heaviside function are examples of activation functions.

[0065] As another example, each neuron is also capable of applying, in addition, a multiplicative factor, also called bias, to the output of the activation function, and the value delivered at the output of said neuron is then the product of the bias value and the value from the activation function.

[0066] The set of qualified data used for training the neural network is data which is, for example, derived from emotion detection tests with a sample of a plurality of users 20. The quantity of tests is, for example, representative of the gestures and therefore of the emotions which can be detected.

[0067] The operation of the determination system 10 according to the invention will now be described with regard to [Fig.2] representing a flowchart of the method for determining the emotion of a user, from a predefined set of emotions.

[0068] The method for determining the emotion of a user comprises a step 100 of acquiring an image (not shown) of the user 20.

[0069] At the end of the acquisition step 100, the method moves on to a next step 110 of estimating a detection confidence score for each emotion, from a predefined set of emotions, respective to each acquired image.

[0070] For example, the predefined set of emotions is for example a group comprising the following emotions: neutral, joy, surprise, sadness, anger, and disgust.

[0071] The detection confidence score is for example a percentage of detection confidence of each emotion. Any known method for performing such an estimation of a detection confidence score for each emotion is applicable, for example the method described in the article entitled “Emotion detection using facial landmarks and deep learning” Rishi Swethan et al, Medium (2018), published on the Internet.

[0072] Preferably, the estimation of the emotion detection confidence score is carried out for each of the emotions of the predefined set.

[0073] The detection confidence score for each emotion is, for example, calculated as a percentage. The percentage is zero percent when the processing unit 22 judges that the emotion to be detected is not at all present among the emotions felt by the user 20. The percentage is one hundred percent when the processing unit 22 is convinced that the emotion to be detected is present among the emotions felt by the user 20, as shown in [Fig. 3] by way of example.

[0074] [Fig.3] illustrates as an example, in table form, the detection confidence scores calculated for each emotion from an image acquired for a given user.

[0075] As appears in the example of [Fig.3], from an acquired image, several emotions can be detected with a fairly high confidence score. This is for example the case, in the example of [Fig.3], of sadness which appears with a detection confidence score of 80% and surprise which appears with a detection confidence score of 40%. Thus, the confidence scores calculated image by image are not sufficient to determine a stable emotion of the user.

[0076] The method comprises repeating the acquisition 100 and estimation 110 steps on successive acquired images.

[0077] For example, the acquisition 100 and estimation 110 steps are carried out at a frequency substantially equal to thirty acquisitions and estimations per second.

[0078] The method then comprises a step 120 of calculating a set of stable emotions over the duration T1, from the detection confidence scores of each emotion for the plurality of acquired images, preferably at least ten acquired images.

[0079] In one embodiment, the duration T1 is 5 seconds with an accuracy of ±5%, preferably the duration is 2 seconds with an accuracy of ±5%, and more preferably the duration is 1 second with an accuracy of ±5%.

[0080] In one embodiment, the calculation of the set of stable emotions over the duration T1 is based on a detection confidence score, estimated in the estimation step 110, greater than a predefined threshold on the set of detection confidence scores of each emotion for the plurality of acquired images.

[0081] For example, the predefined threshold is preferably between 0% and 100%, preferably the predefined threshold is equal to 50%.

[0082] Thus, for example, only emotions having a detection confidence score higher than the predefined threshold over the entire duration T1 are retained in the set of stable emotions, emotions having a detection confidence score lower than the predefined threshold punctually being discarded.

[0083] Following the calculation step 120, the method comprises a step 130 of determining a predominant emotion of the user over a duration T2, from the calculated stable emotions and at least one piece of data from the vehicle 38. The step 130 of determining a predominant emotion implements the artificial intelligence subsystem 25, previously trained to determine a predominant emotion from a set of stable emotions and at least one piece of data from the vehicle 38.

[0084] The at least one piece of data of the vehicle 38 is chosen from the following information: traffic information in an environment of the vehicle, geolocation of the vehicle, internal temperature of the vehicle, external temperature of the vehicle, time, and history of the vehicle's journeys.

[0085] Of course, it is possible to enrich the at least one piece of data of the vehicle 38 to be taken into consideration, for example, external cameras of the vehicle, presence sensors on the seats of the vehicle.

[0086] As a variant, the step 130 of determining the predominant emotion of the user over the duration T2, by the artificial intelligence subsystem 25, is carried out from the calculated stable emotions, from at least one data item of the vehicle 12 and from at least one data item of the user 20.

[0087] In this variant, the artificial intelligence subsystem 25, previously trained to determine a predominant emotion from a set of stable emotions, the at least one piece of vehicle data 38 and the at least one piece of user data 42.

[0088] The at least one user data item is chosen from the following information: electronic calendar, birthday, gender, address, mobile phone notifications, and call history.

[0089] Of course, it is possible to enrich the user's data set to be taken into consideration, with the user's authorization, for example family events.

[0090] The method further comprises a step 140 of transmitting the predominant emotion to a service, for example also implemented by the on-board computer 18 of the vehicle 12, in connection with other equipment (not shown) of the vehicle 12, such as the loudspeakers, the passenger compartment lighting system, a display screen on the dashboard, vibration systems associated with the seats of the vehicle 12.

[0091] By way of example, the steps of determining 130 and transmitting 140 the predominant emotion of the user present in the vehicle 12 are carried out continuously during an activation duration T3 of the system.

[0092] The activation duration T3 of the system corresponds for example to the duration between the acquisition step 100 and the transmission step 140.

[0093] Preferably, the method comprises an additional step 150 of delivering at least one personalized service for the user 20 using at least one piece of equipment of the vehicle, as a function of the predominant emotion. More preferably, the personalized service for the user 20 is as a function of the predominant emotion and is carried out continuously during the activation duration T3 of the system.

[0094] For example, the at least one personalized service is chosen from the following services: type of music, guidance, suggestion of an advertisement, lighting, seat position, connection to the Internet and to services, connection to a smartphone, and massage.

[0095] For example, depending on the predominant emotion, relaxing music or more rhythmic music is automatically selected and played through the speakers.

[0096] Advantageously, the determination method is an automatic determination method.

[0097] Advantageously, the determination method according to the invention, the system, and the associated computer program, make it possible to determine a predominant and precise emotion of the user.

Claims

Claims

1. Method for determining an emotion of a user (20) present in a vehicle, from among a predefined set of emotions, implemented by a determination system (10), the determination system (10) comprising a camera (16) connected to an on-board computer (18), the camera (16) being configured to capture images of said user (20), the on-board computer (18) comprising a processing unit (22), the method being characterized in that it comprises the following steps implemented by the processing unit (22): a. acquisition (100) of a plurality of images of the user (20) over a duration T1, b. estimation (110) of a detection confidence score of each emotion, from among the predefined set of emotions, respective to each acquired image, c. calculation (120) of a set of stable emotions over said duration Tl, from the detection confidence scores of each emotion for said plurality of acquired images, d.determination (130), by an artificial intelligence subsystem (25), of a predominant emotion of the user (20) over a duration T2, from the calculated stable emotions and at least one piece of data from the vehicle (38), the duration T2 being at least equal to the duration T1, and e. transmission (140) of the determined predominant emotion to at least one service using the determined predominant emotion.

2. Method according to claim 1, in which it comprises an additional step (150) of delivering at least one personalized service for the user (20) using at least one piece of equipment of the vehicle, depending on the predominant emotion.

3. Method according to one of claims 1 or 2, in which the at least one vehicle data (38) is chosen from the following information: traffic information in an environment of the vehicle, geolocation of the vehicle, internal temperature of the vehicle, external temperature of the vehicle, time, and history of the vehicle's journeys.

4. The method of any one of claims 1 to 3, wherein the step of determining (130) the determined predominant emotion of the user (20) over the duration T2, by the artificial intelligence subsystem, is carried out from the calculated stable emotions, from at least one vehicle data item and from at least one user data item (42).

5. Method according to claim 4, in which the at least one user data (42) is chosen from the following information: electronic diary, birthday, gender, address, mobile phone notifications, and call history.

6. Method according to any one of claims 1 to 5, in which the calculation (120) of the set of stable emotions over said duration T1 is based on a detection confidence score greater than a predefined threshold on the set of detection confidence scores of each emotion for the plurality of acquired images.

7. A method according to any one of claims 1 to 6, wherein the artificial intelligence subsystem (25) implements a neural network previously trained by deep learning with a qualified data set.

8. System (10) for determining an emotion of a user (20) present in a vehicle, from a predefined set of emotions, the system (10) comprising a camera (16) connected to an on-board computer (18), the camera (16) being configured to capture images of said user (20), the on-board computer comprising a processing unit (22), characterized in that the processing unit (22) is configured to implement: - an acquisition module (26) of a plurality of images of the user (20) over a duration T1, - an estimation module (28) of a detection confidence score of each emotion, from the predefined set of emotions, respective to each acquired image, - a calculation module (30) of a set of stable emotions over said duration T1, from the detection confidence scores of each emotion for said plurality of acquired images, - a determination module (32), by an artificial intelligence subsystem,of a predominant emotion of the user (20) over a duration T2, from the calculated stable emotions and at least one data item provided by the vehicle or by the vehicle and a user (20) of the vehicle, the duration T2 being at least equal to the duration T1; - a transmission module (34) of the predominant emotion de-, completed at least one service using the determined predominant emotion.

9. A computer program comprising executable code instructions, which, when executed by a programmable system, implement a method for determining an emotion of a user according to claims 1 to 7.

Images