Affect enrollment and verification

Abstract

An apparatus for assessing affect state of an occupant of a vehicle includes a sensor that collects information indicative of said occupant's affect display, an enrollment database that comprises affect information derived from said occupant, and an infotainment system that executes an affect assessor.

Description

AFFECT ENROLLMENT AND VERIFICATIONRelated Applications

[0001] This application claims the benefit of the February 9, 2024 filing date of U.S. Provisional Application No. 63 / 551,576, the content of which is hereby incorporated by reference in it entirety.Background

[0002] Many vehicles now include an infotainment system that runs various applications. Among these applications is an “automotive assistant.”

[0003] To improve overall travel experience, many automotive assistants are able to identify an occupant of the vehicle and to load that occupant’s “user preferences.” This allows the automotive assistant to customize its operation of the vehicle in that occupant’s preferences.

[0004] A human occupant has an “affect” that changes state. This affect encompasses an occupant’s emotion, which tends to be transient, and mood, which tends to be less transient.

[0005] Although an affect state is experienced only internally, human beings manifest their affect state through “affect displays.” As a result, it is possible to infer an occupant’s affect based on these affect displays. Examples of affect display include changes in physiognomy, gestures, body language, tone of voice, the semantic content of speech, as well as particularly obvious clues, such as presence of and intensity of laughter or weeping. These affect displays are collectively referred to herein as the “observables.”Summary

[0006] In a growing number of vehicles, the infotainment system executes an application that infers an occupant’s affect state based on observables. This information concerning the occupant’s current affect state is available to other applications that execute on the infotainment system, including the automotive assistant. An application that has access to this information can then tailor its operation to suit the occupant’s affect state.

[0007] In known systems, information about an occupant’s affect state is generated on the fly and then discarded. The invention is based on the recognition that this information can be used for other purposes.

[0008] According to the invention, it is possible to associate information about affect state with a particular occupant, thus improving the accuracy with which that occupant’s emotional state can be ascertained.

[0009] Conventional methods for inferring affect state use a mapping that assigns, to each observable, an affect state. This mapping is typically derived from observation of a large population. An assumption built into this is that the occupant’s observables map to an affect state in the same way that an average person’s observables will map to an affect state. As a result, this procedure will be accurate only on the average. It is not tuned to any particular occupant or small group of occupants.

[0010] In contrast, the affect assessor described herein compares an observed affect state with an occupant’s specific affect baseline. The affect assessor makes use of an existing paradigm used for biometric authentication, and in particular, having an occupant provide occupant-specific data for later use. In the case of biometric authentication, the later use is authentication. In contrast, in the case of affect assessment, the later use is to use an occupant’s baseline affect displays to more accurately assess the occupant’s current affect state. This turns out to be surprisingly useful because the variance in affect display for a given affect state is significant enough so that systems that fail to account for it have a tendency to be error prone.[Oil] As used herein, a “mapping” is an operation that assigns, to each element of a first set, at least one element in a second set, wherein the second set is either the same as or different from the first set. In this case, the “first set” is a set of observables and the “second set” is a set of affect states.

[0012] For example, there exists a set of persons who wear a scowl as their neutral expression. A conventional emotion assessor operating in the manner described above would thus tend to misclassify the emotional state of a person from that set as being discontent. The automotive assistant will then erroneously treat that person as if he were discontent.

[0013] The invention contemplates accommodating the possibility that the mapping from an observables to an emotional state for a particular person may differ significantly from the mapping from an observables to an emotional state for an average person.

[0014] These and other features of the invention will be apparent from the following detailed description and the accompanying figure, in which:Description of Drawings

[0015] FIG. 1 shows an exemplary architecture for a system that reduces accidental triggering of a speech interface.Detailed Description

[0016] FIG. 1 shows a vehicle 10 having an occupant 12. The vehicle 10 includes an infotainment system 14 that executes applications, of which two are shown: an automotive assistant 16, an affect assessor 18, and a biometric identifier 20, all of which are in data communication with an enrollment database 22.

[0017] The enrollment database 22 includes information concerning numerous potential occupants, including the illustrated occupant 12. This information includes occupant-specific information including user preferences 24, biometric information 26, and affect information 28.

[0018] While it is perhaps not readily apparent, the biometric information 26 and the affect information 28 play similar roles. Biometric information 26 for a particular occupant 12 is useful for showing how that occupant differs from an average occupant. Indeed, this is what makes biometric information 26 useful for identifying a particular person.

[0019] Similarly, affect information 28 is useful for showing how that occupant’s affect display differs from the norm. In this case, the differential between the occupant’s affect display and that of the general population is what makes it possible for the affect assessor 18 to more accurately determine the affect state of the occupant.

[0020] The biometric identifier 20 and the affect assessor 18 receive information from a sensor 30. The sensor 30 provides information on one or more observables. Accordingly, the sensor 30 includes: a camera 32, a microphone 34, a haptic sensor 36, a galvanic sensor 38, and / or a chemical sensor 40.

[0021] In operation, the biometric identifier 20 identifies the occupant 12 using the biometric information 26 in the enrollment database 22. It provides this information to both the automotive assistant 16 and the affect assessor 18.

[0022] Using the information about the occupant’s identity, the automotive assistant 16 retrieves the correct user preferences 25 and the affect assessor 18 retrieves the correct affect information 28.

[0023] The affect assessor 18 then uses the affect information 28 to determine the occupant’s affect state, which it then provides to the automotive assistant 16. Examples of such a determination include offsetting or biasing the result that would have been obtained by mapping observables based on a norm from the general population.

[0024] As a result of the foregoing activity, the automotive assistant 16 knows who the occupant 12 is, knows what the occupant’s preferences are, and knows how to tune those preferences based on the occupant’s affect state. Armed with all this information, the automotive assistant communicates with the occupant 12 through an output interface 42, which includes a loudspeaker 44 and / or a display 46.

[0025] Information concerning an occupant’s affect state offers numerous practical applications. For instance, if the haptic sensor detects a crushing grip on the steering wheel, the galvanic sensor detects high conductivity, and the camera observes a grimace and pockets of foam at the mouth’s comers, and if the vehicle is being driven fast, it may be useful to elevate a level of alert and to take steps to modify the occupant’s affect state.

[0026] In some embodiments, the affect assessor 18 receives a waveform that represents an interaction of the occupant with the environment. Examples of interaction include production of sound waves by the occupant, scattering of light incident on the occupant, and emission of radiation by the occupant, and in particular, infrared radiation. In some embodiments, the waveform is a two-dimensional waveform that provides a frequency-domain representation of the interaction. Examples of such waveforms include those that provide, at each time from a set of times, amplitudes and / or phases of frequency components that represent an interaction of the occupant with the environment.

[0027] Some embodiments feature signal-processing circuitry that interacts with the waveform, for example by varying power at each frequency component thereof or by changing an angle of a phasor associated with the waveform. In some embodiments, the signal-processing circuitry, at each time from a set of times, integrates power across a range of frequencies and weights the resulting integral by an inverse of the extent of that range.

[0028] The affect assessor 18 further includes a classifier circuit that receives the waveform. The classifier circuit typically includes sets of nodes that are placed in serieswith each other. Each node in a set of nodes features a summation block that receives a set of inputs. Each input passes through a weighting circuit that applies a suitable gain to the input. The weighted inputs are then summed. Based on the value of the sum, and in particular, whether it exceeds a threshold, the node provides an output. The output of each node in a set of nodes becomes an input to a node in a subsequent set of nodes. The process of selecting the correct gains to apply to each input of each node is determined during a calibration procedure that is carried out before the classification circuit’s actual deployment. The calibration procedure involves associating features of an input to the classifier with various affect states and adjusting the gains of each input of each node based on the resulting associations. A classifier circuit, suitable calibrated in this manner, implements a trained emotion, or affect classifier.

[0029] An occupant’ s emotional state is modeled as a superposition of primary emotions. In such cases, each utterance by an occupant results in a vector of scores, with each element of the vector corresponding to a particular one of these primary emotions. By collecting plural vectors for an occupant, it becomes possible to ascertain an “average emotion” vector in which each element is an average score for a corresponding one of the primary emotions. As a result, it becomes possible to ascertain a threshold vector for that occupant in which each element of the threshold vector is a threshold score for a corresponding one of the primary emotions. In some embodiments, This average-emotion vector and the corresponding threshold vector can then be provided as an input to the classifier circuit to be used for determining the occupant’s emotional state. As a result, the determination of the occupant’s emotional state is based on the occupant’s average emotional state rather than that of a general population.

[0030] Having described the invention and a preferred embodiment thereof, what is claimed as new and secured by letters patent is:

Claims

CLAIMS1. An apparatus for assessing affect state of an occupant of a vehicle, said apparatus comprising a sensor that collects information indicative of said occupant’s affect display, an enrollment database that comprises affect information derived from said occupant, and an infotainment system that executes an affect assessor, said affect assessor being configured to receive said information from said sensor and to use said affect information in said enrollment database in conjunction with said information indicative of said occupant’s affect display to assess said affect state of said occupant.

2. The apparatus of claim 1 , wherein said affect information comprises information that maps an affect display of said occupant to an affect state of said occupant3. The apparatus of claim 1, wherein said sensor collects biometric information from said occupant, wherein said occupant is a first occupant of a set of occupants that includes a second occupant and said first occupant, wherein said enrollment database comprises affect information from said first occupant and from said second occupant, wherein said apparatus further comprises a biometric identifier that uses said biometric information to determine an identity of said occupant, said identity being that of said first occupant, wherein said biometric identifier provides said affect assessor with said identity, and wherein said affect assessor uses said affect information from said first occupant and ignores said affect information from said second occupant.

4. The apparatus of claim 1, further comprising an automotive assistant executed by said infotainment system, wherein said enrollment database comprises user preferences, and wherein said automotive assistant operates using said affect state and said user preferences.

5. The apparatus of claim 1, wherein said affect information derived from said occupant represents a mapping between affect display by said occupant and affect state of said occupant.

6. The apparatus of claim 1 , wherein said affect information derived from said occupant represents a difference between a mapping from affect display by said occupant and affect state of said occupant and a mapping between affect display by said occupant and an affect state of an average person from a population.

7. The apparatus of claim 1, wherein said sensor comprises a camera.

8. The apparatus of claim 1, wherein said sensor comprises a microphone.

9. The apparatus of claim 1, wherein said sensor comprises a haptic sensor.

10. The apparatus of claim 1, wherein said sensor comprises a galvanic sensor.

11. The apparatus of claim 1 , wherein said sensor comprises a chemical sensor.

12. The apparatus of claim 1, further comprising an automotive assistant that executes on said infotainment system, wherein said vehicle further comprises a loudspeaker, wherein using said loudspeaker, said automotive assistant communicates with said occupant in a manner that depends on said affect state of said occupant.

13. The apparatus of claim 1, further comprising an automotive assistant that executes on said infotainment system, wherein said vehicle further comprises a display, wherein using said loudspeaker, said automotive assistant communicates with said occupant in a manner that depends on said affect state of said occupant.

14. The apparatus of claim 1, further comprising an automotive assistant that executes on said infotainment system, wherein said vehicle further comprises a haptic output, wherein using said haptic output, said automotive assistant communicates with said occupant in a manner that depends on said affect state of said occupant.

15. The apparatus of claim 1, further comprising an automotive assistant that executes on said infotainment system, wherein said vehicle further comprises a heater, wherein using said heater, said automotive assistant communicates with said occupant in a manner that depends on said affect state of said occupant.

16. The apparatus of claim 1 , further comprising an automotive assistant that executes on said infotainment system, wherein said vehicle further comprises a cooling device, wherein using said cooling device, said automotive assistant communicates with said occupant in a manner that depends on said affect state of said occupant.

17. The apparatus of claim 1, further comprising an automotive assistant that executes on said infotainment system, wherein said vehicle further comprises a vibrator coupled to a seat, wherein using said vibrator, said automotive assistant communicates with said occupant in a manner that depends on said affect state of said occupant.

18. The apparatus of claim 1, wherein said occupant is a first occupant of a set of occupants that includes a second occupant and said first occupant, wherein said enrollment database comprises affect information from said first occupant and from said second occupant, wherein said apparatus further comprises an identifier that determines an identity of said occupant, said identity being that of said first occupant, wherein said identifier provides said affect assessor with said identity, and wherein said affect assessor uses said affect information from said first occupant and ignores said affect information from said second occupant.

19. The apparatus of claim 1, wherein said occupant is a first occupant of a set of occupants that includes a second occupant and said first occupant, wherein said enrollment database comprises affect information from said first occupant and from said second occupant, wherein said apparatus further comprises an identifier that determines an identity of said occupant based on credentials provided by said occupant, said identity being that of said first occupant, wherein said identifier provides said affect assessor with said identity, and wherein said affect assessor uses said affect information from said first occupant and ignores said affect information from said second occupant.

20. The apparatus of claim 1 , wherein said occupant is a first occupant of a set of occupants that includes a second occupant and said first occupant, wherein saidenrollment database comprises affect information from said first occupant and from said second occupant, wherein said apparatus further comprises an identifier that determines an identity of said occupant based on information provided by a portable electronic device carried by said occupant, said identity being that of said first occupant, wherein said identifier provides said affect assessor with said identity, and wherein said affect assessor uses said affect information from said first occupant and ignores said affect information from said second occupant.

Images