Time-varying countenance assessment for affect state estimation

Abstract

A method includes generating an embedding vector for each image among a plurality of images of an occupant in a vehicle. The plurality of images define an image sequence. The method further includes defining an ordered sequence of the embedding vectors, and outputting an emotional state of the occupant based at least in part on an assessment of the ordered sequence of the embedding vectors using a neural network. The ordered sequence being indicative of temporal sequence of the plurality of images.

Description

TIME-VARYING COUNTENANCE ASSESSMENT FOR AFFECT STATE ESTIMATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. provisional application Serial No. 63 / 661,090 filed June 18, 2024, the disclosure of which is hereby incorporated in its entirety by reference herein.TECHNICAL FIELD

[0002] The present disclosure is generally directed to a system and / or method for detecting emotion of an occupant in a vehicle.BACKGROUND

[0003] A human face is often regarded as a window into that human’s emotional state. For example, when one sees an image of a human with a mirthful expression, one reasonably infers that the person is full of mirth. An image of a person with a tearstained face and a smile suggests one who is in a state of uncontrolled laughter.SUMMARY

[0004] In some aspects, the present disclosure is directed to a method including: generating an embedding vector for each image among a plurality of images of an occupant in a vehicle, the plurality of images defining an image sequence; defining an ordered sequence of the embedding vectors, the ordered sequence being indicative of temporal sequence of the plurality of images; and outputting an emotional state of the occupant based at least in part on an assessment of the ordered sequence of the embedding vectors using a neural network.

[0005] In other aspects, the present disclosure is directed to a system for assessing emotional state of an occupant of a vehicle. The system includes a processor and a non-transitory computer readable medium comprising programming instructions. When executed by the processor, theprogramming instructions cause the processor to: generate an embedding vector for each image among a plurality of images of the occupant using an embedding model, the plurality of images defining an image sequence; define an ordered sequence of the embedding vectors, the ordered sequence being indicative of temporal sequence of the plurality of images; and output an emotional state of the occupant based at least in part on an assessment of the ordered sequence of the embedding vectors using a classifier model.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 shows a vehicle having a countenance assessor with an infotainment system of the vehicle;

[0007] FIG. 2 shows details of the countenance assessor of FIG. 1; and

[0008] FIG. 3 shows assessment of a static countenance.DETAILED DESCRIPTION

[0009] As required, detailed embodiments of the present invention are disclosed herein; however, it is to be understood that the disclosed embodiments are merely exemplary of the invention that may be embodied in various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present invention.

[0010] In some cases, the manifestation of an emotional state begins with an onset, followed by a peak display of emotion, and ending with an asymptotic approach to a neutral emotional state. This evolution of one’s countenance over time may provide considerable information on the emotional state of the person. By way of analogy, a phrase such as “I am so angry” conveys different emotion depending on which of the words is accented. Thus, if one evaluates emotion at each word and then takes an average, the result is likely to be inaccurate.

[0011] A computer-based emotion recognition system may analyze, or stated differently, measure features of an image of a face to determine an emotion of the person. In some aspects, multiple images of the person’s face is measured and then an average of measurements of the images is used to determine the emotion of the person.

[0012] The present disclosure is generally directed to a system / method that takes into account the evolution of a person’s countenance in evaluating an emotional state of the person. In one aspect, the system / method processes a video of a person in a manner that causes it to yield information about that person’s emotional state. For example, the process includes processing each image of a sequence of images of the video using a first neural network to yield respective numerical characterizations of those images. The numerical characterizations are further processed using a second neural network that yields information regarding the person’s emotional state.

[0013] In some aspects, each numerical characterization comprises more elements than there are distinct emotions represented in the emotion determination.

[0014] Processing the numerical characterizations may include assembling an array that represents the numerical characterizations and processing the array using the second neural network. In some aspects, dimensions of the array may depend on both the number of images and the number of elements in each numerical characterization. In some aspects, the second neural network comprises a convolutional neural network (CNN).

[0015] In some variations, the dimensions of the array depends only on the number of elements in each numerical characterization. In such variation, if the number of images falls short of what may be needed to fill the array, the array is padded to an extent that is necessary to make up the difference.

[0016] Referring to FIG. 1, a vehicle 10 includes an infotainment system 12 having, among other components, an application module 13 that executes software applications 14. Among the applications 14 is a countenance assessor application 16 (“countenance assessor 16” in FIG. 1) that is configured to detect an emotional state of an occupant 18 of the vehicle 10 based on an image sequence 20 provided by a camera 22 that is directed at the occupant 18. In someimplementations, the application module 13 includes, among other components, one or more computing devices configured to store and execute the applications 14 including the countenance assessor application 16. In another implementation, the countenance assessor application 16 may be provide as a countenance assessor module including, at least, a dedicated processor and a dedicated non- transitory computer readable medium storing the application 16 that is executed by the processor. The countenance assessor module may be integrated with the infotainment system 12.

[0017] FIG. 2 illustrates the countenance assessor 16 receiving an image sequence 20 of the occupant 18 via the camera 22. The image sequence 20 comprises multiple images 24, each of which was acquired during a particular time interval. The camera 22 operates at a particular frame rate that controls how many images 24 are acquired within a particular time interval. A typical camera 22 will acquire, during each second of image acquisition, approximately thirty such images 24.

[0018] Each image 24 is separated from its neighboring image by a reciprocal framerate. The image sequence 20 provides information indicative of the evolution of the occupant’s facial configuration during an interval. A duration of the interval is approximated as a product of the cardinality of the image sequence 20 and the reciprocal framerate. The evolution of countenance during the interval includes: an onset; a peak, during which the emotion is at its maximum; and an offset, during which the display of peak emotion dissipates.

[0019] In the course of operation, the countenance assessor 16 observes the evolution of the occupant’s countenance during an observation interval. The length of the observation interval depends on various biophysical parameters, such as the rate at which action potentials propagate and the rate at which facial muscles are able to respond to those action potentials. However, for most human occupants 18, more than enough useful information concerning evolution of countenance can be found within about two seconds from the onset of an emotional state. As a result, the image sequence 20 may have approximately sixty images 24.

[0020] In some aspects, each image 24 has a high dimensionality. For example, it is not unusual for the image 24 to have three hundred or more pixels on a side. As a result, it is useful to providean embedding model (EM) 26 that is configured to generate, for each image 24, a corresponding embedding vector 28 in a lower-dimensional space. In a non-limiting example, the embedding model 26 includes a neural network such as, but not limited to, a convolutional neural network (CNN) or a residual neural network (RNN). The image 24 passes through the neural network of the embedding model 26 to generate the embedding vector 28.

[0021] A sequencer model 30 receives the embedding vectors 28 and is configured to organize the vectors 28 into an ordered sequence that reflects a temporal order of the images 24 from which they were derived. As a result, the ordered sequence preserves information about the evolution of the occupant’s facial configuration. In a non-limiting example, the sequencer may be provided as recurrent neural networks (RNN), long short-term memory (LSTM), gated recurrent unit (GRU), or a diffusion module, among others.

[0022] The sequencer model 30 provides the ordered sequence to a classifier model 32 that has been trained to provide a score at each output line 34 in an output array 36. The score from the classifier model 32 represents a classification of the emotional state of the occupant 18. Stated differently, the classifier model 32 recognizes a plurality of defined emotional states, where each emotional state is associated with a different value that is provided as the score. When assessing the ordered sequence, the scores of the output lines 34 is associated with emotional states. For example, the classifier model 32 is configured to assign the score(s) reflective of the determined emotion based on the ordered sequence of the embedded vectors. As a result, the countenance assessor 16 carries out a single classification step that takes into account the entire evolution of the occupant’s countenance as recorded in the image sequence 20. In a non-limiting example, the classifier model 32 includes a neural network, such as but not limited to: a CNN, including one that takes a variable amount of input and produces a fixed-length representation thereof; sequence models; RNNs; or LSTM neural networks.

[0023] In some cases, the countenance assessor 16 classifies the occupant 18 as being in one emotional state provided among a set of enumerated emotional states based on the ordered sequence of the embedding vectors. For example, the classifier model 32 is configured to analyzethe scores associated with the set of enumerated emotional states and selects the emotional state having the highest score as the emotional state of the occupant 18.

[0024] Other embodiments of the countenance assessor 16 take a more nuanced view. For example, the countenance assessor 16 is configured to treat each emotional state in the set of enumerated emotional states as a basis vector in an emotional space and classify the emotional state of the occupant based on affect vector detemiined by weighting the enumerated emotional states.

[0025] In some embodiments, training the countenance assessor 16 includes training the classifier model 32 and the embedding model 26 together. Embodiments of such joint training of the classifier model 32 and the embedding model 26 include those in which training data is optional but nevertheless useful as an initial estimate and those in which it is needed for the entire procedure.

[0026] In some aspects, referring to FIG. 3, a static model 38 that has been trained to ascertain emotional state based on a static countenance (e.g., a single image) may be used as the embedding model 26. For example, the static model 38 may include an input layer 40, an output layer 42, and intermediate layers 44, each of which outputs an activation that is to be provided to its succeeding layer with an assumption being that the density of useful information in an activation increases with each succeeding layer. The static model 38 that has been truncated at a particular layer 46 of the intermediate layers 44 provides its latest actuation to the sequencer model 30 instead of the layer that succeeds the particular layer 46. In this case, only the classifier model 32 has to be trained.

[0027] In some embodiments, the countenance assessor 16 relies on a combination of audio and video to assess the occupant’s emotion. In such embodiments, the countenance assessor 16 also receives a spectrogram of the occupant’s speech as it occurred during the time spanned by the image sequence 20. This spectrogram is then provided to the countenance assessor 16 along with the image sequence 20. The countenance assessor 16 either considers the spectrogram separately from the images. In other embodiments, the spectrogram is provided to the sequencer model 30, which then incorporates it with the image sequence 20.

[0028] While exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms. Rather, the words used in the specification are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the present disclosure. Additionally, the features of various implementing embodiments may be combined to form further embodiments.

[0029] In this application, the term “module” (e.g., the application module 13 having the countenance assessor 16, or a countenance assessor module) may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on- chip.

[0030] The term memory is a subset of the term computer-readable medium. The term computer- readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory. Non-limiting examples of a non- transitory, tangible computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read only circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

[0031] The systems, modules, and / or models (e.g., the application module 13, the countenance assessor application 16, the embedding model 26, the sequencer model 30, and / or the classifier model 32) described in this application may be partially or fully implemented by a special purpose computer created by configuring a general-purpose computer to execute one or more particular functions embodied in computer programs. The functional blocks, flowchart components, andother elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.

[0032] As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”

[0033] The description of the disclosure is merely exemplary in nature and, thus, variations that do not depart from the substance of the disclosure are intended to be within the scope of the disclosure. Such variations are not to be regarded as a departure from the spirit and scope of the disclosure

Claims

WHAT IS CLAIMED IS:

1. A method comprising: generating an embedding vector for each image among a plurality of images of an occupant in a vehicle, the plurality of images defining an image sequence; defining an ordered sequence of the embedding vectors, the ordered sequence being indicative of temporal sequence of the plurality of images; and outputting an emotional state of the occupant based at least in part on an assessment of the ordered sequence of the embedding vectors using a neural network.

2. The method of claim 1 , wherein the neural network is a convolutional neural network trained to output the emotional state of the occupant.

3. The method of claim 1, wherein the emotional state of the occupant is outputted based on an assessment of audio data in addition to the assessment of the ordered sequence, the audio data being indicative of sounds by the occupant during an interval defined by the image sequence.

4. The method of claim 3 wherein the audio data is provided with the image sequence.

5. The method of claim 1, further comprising passing each image through a neural network to generate the embedding vector.

6. The method of claim 5, wherein the neural network is one of a convolutional neural network or a residual neural network.

7. The method of claim 1, wherein the assessment of the ordered sequence of the embedding vectors comprises assessing the emotional state based at least in part on a rate of change between the images.

8. The method of claim 1, further comprising, for the assessment of the ordered sequence of the embedding vectors: assigning a plurality scores to the ordered sequence of the embedding vectors; and classifying the occupant as being in the emotional state associated with the score that is highest among the plurality of scores assigned to the ordered sequence of the embedding vectors.

9. The method of claim 1, further comprising, for the assessment of the ordered sequence of the embedding vectors: assigning a plurality scores to the ordered sequence of the embedding vectors; and classifying the occupant as being in the emotional state associated with a final score determined by a combination of the plurality of scores.

10. A system for assessing emotional state of an occupant of a vehicle, the system comprising: a processor; and a non-transitory computer readable medium comprising programming instructions that when executed by the processor, cause the processor to: generate an embedding vector for each image among a plurality of images of the occupant using an embedding model, the plurality of images defining an image sequence; define an ordered sequence of the embedding vectors, the ordered sequence being indicative of temporal sequence of the plurality of images; and output an emotional state of the occupant based at least in part on an assessment of the ordered sequence of the embedding vectors using a classifier model.

11. The system of claim 10, wherein the classifier model and the embedding model include neural networks that are jointly trained.

12. The system of claim 10, wherein the classifier model and the embedding model include neural networks that have been trained separately.

13. The system of claim 10, wherein the programming instructions further cause the processor to assess, using the classifier model, the emotional state based at least in part on a spectrogram and the sequence of embedding vectors, the spectrogram being indicative of an audio data that extends for a duration that overlaps the image sequence.

14. The system of claim 10, wherein the classifier model includes a neural network.

15. The system of claim 10, wherein the classifier model includes one of a convolutional neural network, recurrent neural network, along short-term memory neural network.

16. The system of claim 10, wherein the embedding model includes a neural network.

17. The system of claim 10, wherein the embedding model includes one of a convolutional neural network or residual neural network.

18. The system of claim 10, wherein the programming instructions for the assessment of the ordered sequence of the embedding vectors further cause the processor to assess the emotional state based at least in part on a rate of change between the images.

19. The system of claim 10, the programming instructions for the assessment of the ordered sequence of the embedding vectors further cause the processor to: assign a plurality scores to the ordered sequence of the embedding vectors; and classify the occupant as being in the emotional state associated with the score that is highest among the plurality of scores assigned to the ordered sequence of the embedding vectors.

20. The system of claim 10, further comprising a camera disposed in the vehicle and operable to capture the plurality of images.

Images