Method and system for measuring shopper response to products based on behavior and facial expression

Abstract

The present invention is a method and system for measuring human response to retail elements, based on the shopper's facial expressions and behaviors. From a facial image sequence, the facial geometry—facial pose and facial feature positions—is estimated to facilitate the recognition of facial expressions, gaze, and demographic categories. The recognized facial expression is translated into an affective state of the shopper and the gaze is translated into the target and the level of interest of the shopper. The body image sequence is processed to identify the shopper's interaction with a given retail element—such as a product, a brand, or a category. The dynamic changes of the affective state and the interest toward the retail element measured from facial image sequence is analyzed in the context of the recognized shopper's interaction with the retail element and the demographic categories, to estimate both the shopper's changes in attitude toward the retail element and the end response—such as a purchase decision or a product rating.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]Not ApplicableFEDERALLY SPONSORED RESEARCH[0002]Not ApplicableSEQUENCE LISTING OR PROGRAM[0003]Not ApplicableBACKGROUND OF THE INVENTION[0004]1. Field of the Invention[0005]The present invention is a method and system to provide an automatic measurement of retail customers' responses to retail elements, based on their facial expressions and behaviors.[0006]2. Background of the Invention[0007]The current consumer and market-oriented economy places a great deal of importance on people's opinions or responses to consumer products or, more specifically, various aspects of the products—product display, packaging, labels, and price. A shopper's interest and attitude toward these elements changes dynamically during engagement and interaction with products, and the end response—such as purchase, satisfaction, etc.—is a final summary of such intermediate changes. Most consumer exposure to such visual cues occurs in retail spaces at an immeasurably...

AI Technical Summary

Benefits of technology

[0025]It is one of the objectives of the first step of the processing to detect faces, track them individually, and estimate both the two-dimensional and three-dimensional poses of each of the tracked faces. Given a facial image sequence, the step detects any human faces and keeps individual identities of them by tracking them. Using learning machines trained from facial pose estimation training, the two-dimensional facial pose estimation step computes the (X, Y) shift, size variation, and orientation of the face inside the face detection window to normalize the facial image, as well as to help the three-dimensional pose estimation. The two-dimensional facial pose estimation training requires facial images having varied two-dimensional geometry—(X, Y) shifts, sizes S, and orientations O—that reflect the variations from the face detection step, along with the ground truth values of these variations. Multiple learning machines are trained, where each machine is trained to output high response to facial images having (X, Y, S, O) close to the predetermined (X0, Y0, S0, O0) of the machine. The three-dimensional facial pose estimation computes the yaw (horizontal rotation) and pitch (vertical rotation) in a manner similar to the two-dimensional facial pose estimation.

Problems solved by technology

Though it is nearly impossible to accurately determine a person's mental response without directly asking about it, a person usually reveals some indications of emotional response through information channels such as facial expressions and bodily gestures.

It is not straightforward to make a direct connection between the emotion-sensitive filter responses and the facial expressions due to the complex relation between the image responses and the expressions; a large number of such emotion-sensitive feature vectors along with the ground truth expression categories are utilized to learn the relation in a machine learning framework.

Images