Fragrance preference degree detection system and method based on facial micro-expression

Abstract

The invention discloses a fragrance preference degree detection system and method based on facial micro-expressions. The system comprises a data acquisition module and a detection module, the data acquisition module is used for creating a test scene and acquiring facial micro-expressions of a subject in a test process to obtain micro-expression data; the detection module is used for constructing feature vectors according to the micro expression data; when a plurality of evaluation steps exist in the test process, respectively constructing the feature vectors according to the evaluation steps; predicting the feature vector through a trained neural network model to obtain a detection result; the method can recognize the facial micro-expression based on the evaluation video, and achieves the detection of the preference degree according to the micro-expression data in the whole evaluation process.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and more particularly to a fragrance preference detection system and method based on facial micro-expressions. Background Art

[0002] At present, traditional consumer product evaluation, especially fragrance experience evaluation, mainly adopts subjective questionnaire survey and sensory evaluation methods. However, these methods have a series of limitations, including strong subjectivity, poor timeliness, and lack of objectivity. Subjective questionnaire survey: Traditional fragrance evaluation methods mainly rely on subjective questionnaires filled out by subjects, which contain subjective evaluations such as overall preference for fragrance and sensory experience. However, this method is easily affected by the subjective will of the subjects and individual subjective differences, and the results lack objectivity. Sensory evaluation instruments: Some advanced fragrance evaluation methods use sensory evaluation instruments, such as odor recognition instruments and physiological sensing instruments. These instruments can quantify certain attributes of fragrance, but still cannot fully capture the emotional experience of consumers, and lack real-time monitoring of individual emotional changes. Physiological response detection: Some studies use physiological response detection, such as heart rate changes, skin electrical response, etc., to indirectly reflect the individual's emotional response to fragrance. However, these physiological indicators are affected by many factors, and it is difficult to accurately reflect the true emotional state of consumers. In addition, they usually require close-fitting monitoring instruments or sensors to collect physiological signals or even cells or blood, which seriously affects the experience. Facial expression analysis: In the field of emotion recognition, there have been studies using facial expression analysis technology, such as facial expression recognition systems based on deep learning. However, these studies mainly focus on the basis of emotion recognition and lack in-depth analysis related to the fragrance experience.

[0003] In summary, although there have been some studies on emotion recognition and fragrance evaluation, there is no comprehensive and real-time evaluation method in the field of combining facial micro-expression analysis technology with fragrance experience.

[0004] Therefore, how to identify the preference for fragrance through micro-expressions is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a fragrance preference detection system and method based on facial micro-expressions, which can recognize facial micro-expressions based on evaluation videos and realize preference detection based on micro-expression data of the entire evaluation process.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A fragrance preference detection system based on facial micro-expressions, comprising a data acquisition module and a detection module;

[0008] The data acquisition module is used to create a test scene and collect the facial micro-expressions of the subjects during the test to obtain micro-expression data;

[0009] The detection module is used to construct a feature vector according to the micro-expression data; when there are multiple evaluation steps in the test process, the feature vector is constructed according to each evaluation step; the feature vector is predicted by a trained neural network model to obtain a detection result.

[0010] Preferably, the data acquisition module includes an information entry submodule, a parameter configuration submodule, an image acquisition submodule and a voice broadcast submodule;

[0011] The information input submodule is used to input basic information of the subject;

[0012] The parameter configuration module is used to configure the first parameter for the image acquisition submodule; and configure the second parameter for the voice broadcast submodule;

[0013] The image acquisition submodule is used to confirm the position and time of image acquisition according to the first parameter;

[0014] The voice broadcast submodule is used to confirm the content and timing of the voice broadcast according to the second parameter.

[0015] Preferably, the data collection submodule further includes a visualization operation submodule, and the visualization operation submodule is used to present the questionnaire and generate survey results.

[0016] Preferably, the feature vector includes valence level, emotion concentration and intensity of multiple emotions.

[0017] A method for detecting fragrance preference based on facial micro-expressions comprises the following steps:

[0018] Get facial videos during tasting;

[0019] Preprocessing the facial video to construct a data set; the preprocessing step includes:

[0020] Sampling facial video samples, extracting facial movements and identifying micro-expressions, after multiple sampling and identification, multiple expression dimensions carrying micro-expression information are obtained accordingly; the micro-expression information of multiple expression dimensions is attribute-merged to confirm the feature values ​​of each expression dimension, and the feature vector corresponding to a single video sample is obtained;

[0021] The detection model is trained by constructing a data set, and the trained detection model is used to detect the preference.

[0022] Preferably, the steps also include: performing data cleaning on the data set to remove outlier data, missing data and duplicate data through data cleaning; and training the model with the cleaned data.

[0023] Preferably, the step of obtaining the facial video during evaluation includes:

[0024] Set the evaluation steps and generate evaluation process prompts;

[0025] Adjusting the camera so that the acquisition area of ​​the camera covers the face of the subject;

[0026] Complete each tasting step according to the prompts of the tasting process, and collect corresponding video data of each tasting step through the camera.

[0027] Preferably, the micro-expression information includes the intensity, valence level and expression concentration of basic emotions.

[0028] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a fragrance preference detection system and method based on facial micro-expressions, which can recognize facial micro-expressions based on evaluation videos and realize preference detection based on micro-expression data of the entire evaluation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0030] Figure 1 A schematic diagram of the structure of a fragrance preference detection system based on facial micro-expressions provided by the present invention;

[0031] Figure 2 Schematic diagram of a neuron model of a detection model in an embodiment of the present invention;

[0032] Figure 3 Schematic diagram of the detection model structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example 1

[0035] The embodiment of the present invention discloses a fragrance preference detection system based on facial micro-expressions, comprising a data acquisition module and a detection module.

[0036] The data acquisition module is used to create the test scene and collect the facial micro-expressions of the subjects during the test to obtain micro-expression data; the detection module is used to construct a feature vector based on the micro-expression data; when there are multiple evaluation steps in the test process, a feature vector is constructed according to each evaluation step; the feature vector is predicted by the trained neural network model to obtain the detection result.

[0037] Among them, the data acquisition module collects video data of the subject in a specific scenario through image acquisition as the original data for micro-expression recognition.

[0038] The specific scenario includes basic facilities for consumer testing and different evaluation steps. Exemplarily, the evaluation steps provided in this embodiment include observing the appearance and smelling the aroma. For the two stages of observing the appearance and smelling the aroma, the data acquisition module marks the collected video according to the current scenario, and the entire data acquisition process is executed by software control.

[0039] In order to further implement the above technical solution, the data acquisition module includes an information entry submodule, a parameter configuration submodule, an image acquisition submodule and a voice broadcast submodule.

[0040] The information entry submodule is used to enter the basic information of the subject; the parameter configuration module is used to configure the first parameter for the image acquisition submodule; and the second parameter for the voice broadcast submodule; the image acquisition submodule is used to confirm the location and time of image acquisition according to the first parameter; and the voice broadcast submodule is used to confirm the content and timing of the voice broadcast according to the second parameter.

[0041] In this embodiment, in order to make the evaluation follow the preset process, the subject is assisted to complete the specific evaluation process and collect the corresponding video samples by means of guidance prompts. The form of guidance prompts needs to be implemented by means of parameter configuration, such as using a broadcasting method for prompts. The parameter configuration module can be used to configure the start and end prompts of each evaluation step or the duration of each evaluation step, and the parameters of image acquisition can also be configured.

[0042] The generation process of a single video sample is as follows: first, the basic information of the current subject, such as name, gender, number, etc., is entered through the information entry submodule, and the image acquisition submodule adjusts the position of the camera to control the acquisition distance and the proportion of the face in the video screen; the voice broadcast submodule plays the first evaluation step "observe the appearance" and then enters the timing, while the image acquisition submodule controls the camera to collect video and marks the collected data to reflect the first evaluation step; then the second evaluation step "smell" is played and then enters the timing, while the image acquisition submodule controls the camera to collect video and marks the collected data accordingly to reflect the second evaluation step, and finally completes the collection of video data.

[0043] Furthermore, in order to train the detection model, a visualization operation submodule is also included to present the survey documents and generate survey results. After each subject completes the evaluation, a score is given through visualization operation, which is used as a label to be put into the model training.

[0044] In order to further implement the above technical solution, for the construction of feature vectors, the detection module first uses the Facial Action Coding System (FACS) to monitor and identify the facial features in the video (including more than 500 key muscle sites such as eyebrows, eyes, nose, upper and lower lips) in real time, and judges 21 types of facial actions by capturing the changes of these muscle sites within 0.2 seconds, including "smile", "wink", "frown", "raise eyebrows (raise forehead)", "wrinkle forehead", "close eyes", "wrinkle nose", "raise lips", "lower lips", "close mouth", "lip suction", "open mouth", "raise chin", "drop chin", "show dimples", "eyes widen", "cheeks raise", "purse lips", "lip pressure", "smirk", etc. Through these actions, the instantaneous micro-expressions of the face in the video can be identified, which are mainly divided into the intensity of eight basic emotions (range: 0-1) such as "excitement", "annoyance", "concentration", "surprise", "excitement", "joy", "sadness", and "calmness" and the intensity of two basic emotional dimensions (range: -100 to 100) of "valence level" and "expression concentration". That is to say, based on the original video, multiple micro-expression information can be extracted through FACS, and the micro-expression information can reflect the emotional changes during the entire evaluation process.

[0045] Regarding facial expression recognition, the main steps include:

[0046] First, a large dataset containing a variety of expression types is needed. The dataset should include expression samples from various cultural backgrounds to ensure that the model has generalization capabilities. Commonly used public datasets such as KDEF (Karolinska Directed Emotional Faces) and RaFD (Radboud Faces Database) provide rich emotional expression samples.

[0047] Data preprocessing: Data preprocessing includes face detection, alignment, and normalization. Face detection is usually performed using the AdaBoost model based on Haar eigenvalues. Once a face is detected, feature point alignment techniques (such as affine transformation based on facial feature points) are used to align and normalize the facial image for subsequent feature extraction.

[0048] Feature extraction: Feature extraction is one of the key steps in model building. Deep learning-based methods, such as convolutional neural networks (CNNs), can automatically extract facial features from images. A typical CNN includes multiple convolutional layers and pooling layers to capture local features in an image, followed by fully connected layers for feature integration.

[0049] Classifier training: Train the classifier to recognize eight types of expressions. In CNN, the last layer is usually a Softmax layer, which is used to output the probability of each expression category. The training process includes forward propagation and back propagation, and the classification error is minimized by gradient descent.

[0050] In this embodiment, the present invention introduces a two-dimensional model of emotions based on the recognition of basic expressions. The two-dimensional model of emotions usually includes two dimensions: valence and arousal, which are used to quantify emotional states. Valence reflects an individual's emotional tendency towards a stimulus, ranging from negative (unpleasant) to positive (pleasant); while arousal measures the intensity or excitement level of emotions, ranging from low arousal (calm) to high arousal (excited). This quantification method can more comprehensively describe the complexity of emotions and provide richer information than a single dimension.

[0051] Valence level: The valence level dimension is determined by analyzing the individual's emotional evaluation of the stimulus. For example, positive emotions such as happiness and love usually have high positive valence, while negative emotions such as sadness and anger have low negative valence. In facial expression recognition, valence can be estimated by identifying changes in facial muscle activity, such as smiling is usually associated with positive valence, while frowning is associated with negative valence.

[0052] Expression intensity: The expression intensity dimension measures the intensity or energy level of an emotion. Emotions with high expression intensity, such as excitement and anger, show strong physiological activation, while emotions with low expression intensity, such as calmness and boredom, show less physiological response. In facial expression recognition, expression intensity can be assessed by detecting the intensity of facial muscle activity. For example, wide eyes and an open mouth are usually associated with high expression intensity, while closed eyes or slight changes in expression are associated with low expression intensity.

[0053] In practical applications, facial expression recognition systems recognize eight basic expressions and further convert them into scores of valence and expression intensity. These scores are usually based on a pre-trained model that can learn the relationship between different expressions and valence and expression intensity from a large amount of labeled data.

[0054] Specifically, for the evaluation of valence level, the facial micro-expression recognition software identifies different types of facial expressions by detecting key points in the face image, and maps them to the numerical values ​​of valence and expression concentration. For example, when the software detects a smile, it will be regarded as a signal with high valence; when a furrowed brow is detected, it will be considered a sign of low valence. Similarly, for the evaluation of expression concentration, wide eyes and an open mouth will be regarded as features of high expression concentration, while closed eyes or a relaxed expression are considered to be manifestations of low expression concentration.

[0055] In general, the emotional intensity of the eight expressions is a classification method, and the expression concentration and valence level are another classification method for facial expressions. The eight expressions divide emotions into eight basic dimensions, and it is believed that these eight emotions cover all facial expressions. The expression concentration and valence level are two-dimensional classification models, and it is believed that all expressions are composed of two combinations from negative to positive, from calm to excited (rich). Therefore, facial micro-expressions can be obtained from these two models. The values ​​of ten dimensions.

[0056] In order to further construct the feature vector, the detection module merges the attributes of the emotions of each micro-expression in the entire evaluation process through preprocessing to reduce the computational cost.

[0057] Specifically, for a certain evaluation stage, the mean values ​​of the intensity of various emotions, valence levels, and expression concentrations in this stage are counted to obtain a feature vector consisting of 10 expression dimensions. Therefore, by collecting videos of the two evaluation steps of observing the appearance and smelling the odor, an observation vector consisting of 20 expression dimensions can be obtained. (1) , x (2) ,x (3) ,...,x (i) ,...,x (m-1) ,x (m) ], where m = 20, x (i)represents the i-th feature; x (1) ~x (10) represents the average score of 10 expression dimensions in the “appearance observation” stage, x (11) ~x (20) Represents the average value of 10 expression dimensions in the “smell the fragrance” stage.

[0058] Furthermore, using x j Represents the jth variable among multiple input observation variables: The comprehensive preference score of the subject for the sample in the sensory questionnaire is used as the output observation value and recorded as y j , Construct a data set: T = {(x1,y1),(x2,y2),...,(x N ,y N )}.

[0059] In order to further implement the above technical solution, data cleaning and standardization steps are also included. For the constructed data set, data cleaning and standardization are performed before inputting into the detection model.

[0060] Data cleaning mainly includes the steps of removing outliers, missing data, duplicate data, etc. Among them, the interquartile range (IQR) method is used to observe and judge outliers.

[0061] The values ​​of each input variable sequence and output observation sequence that are less than Q L -1.5IQR or greater than Q L The value of -1.5IQR is recorded as an outlier, where Q L represents the lower quartile, Q U It represents the upper quartile, and IQR represents the interquartile range, that is, IQR = Q U -Q L Specifically, the upper and lower bounds of the reasonable interval of the observation sequence and the median are marked by visualization software. According to the reasonable interval, the values ​​of the observation sequence outside the upper and lower bounds are judged as outliers and marked with specific symbols. Finally, samples with outliers and missing values ​​are directly eliminated to prevent interference with modeling.

[0062] The standardization process includes normalizing each feature vector and output observation value vector in the constructed data set according to the maximum and minimum value normalization principle:

[0063]

[0064] Where X min is the minimum value of the sample, X max is the sample maximum value.

[0065] In order to further implement the above technical solution, this embodiment uses a feedforward neural network to perform model training on the standardized data set, randomly selecting 80% of the sample data as a training set for model training, and 20% as a test set for model testing.

[0066] The computation process of a neuron in the model is as follows Figure 2 As shown, the input is the feature vector X, and the output is y, y = f(μ+b), where Model parameters is the feature vector weight; model parameter is the bias, and the function f(.) is the activation function. Here, the symbol function sign(x) is used as the activation function, that is,

[0067] For a single node, at the beginning, the feature vector is used as input, the randomly initialized weights and thresholds are connected, the output of the middle layer nodes is calculated, and then the output of the middle layer nodes is used as the input of the output layer nodes to calculate the output of the output layer nodes. The overall neural network design is as follows Figure 3 As shown, for the output of a single neuron, the errors of the intermediate layer and output nodes are calculated, and the objective function E adopts the least squares method (Among them, Y k is the actual output vector, T k is the expected vector, k is the number of training samples, k = 1, 2, ..., N) to determine whether the error is less than the minimum error or whether the number of training times reaches the preset upper limit. If so, stop training, otherwise adjust the weight Δω according to the error correction learning algorithm. ij Adjust, Δω ij =ηδ j Y j , where η is the learning rate, δ j The deviation between the output value and the expected value is calculated. The whole process is repeated continuously until the error is less than the set upper limit or the number of learning times is greater than the set upper limit.

[0068] Example 2

[0069] Based on the same inventive concept, an embodiment of the present invention discloses a method for detecting fragrance preference based on facial micro-expressions, comprising the following steps:

[0070] S1: Obtain facial video during evaluation.

[0071] S2: Preprocess the facial video and construct a dataset; the preprocessing steps include:

[0072] S21: Sampling facial video samples, extracting facial movements and identifying micro-expressions. After multiple sampling and identification, multiple expression dimensions carrying micro-expression information are obtained accordingly.

[0073] S22: merging the attributes of micro-expression information of multiple expression dimensions to confirm the feature values ​​of each expression dimension and obtain a feature vector corresponding to a single video sample.

[0074] S3: Train the detection model using the constructed data set, and use the trained detection model to detect the preference.

[0075] In this embodiment, the facial video during the evaluation records the micro-expression changes of consumers (or subjects) during the evaluation of fragrances. Each consumer corresponds to a video sample. When processing a certain video sample, the FACS system samples the video sample and generates a recognition result accordingly to obtain an expression dimension; each expression dimension describes the micro-expression information by the category of basic emotions and the corresponding intensity.

[0076] In order to further implement the above technical solution, the method of performing attribute merging in S22 is specifically as follows:

[0077] In order to further implement the above technical solution, the steps also include: performing data cleaning on the data set to remove outlier data, missing data and duplicate data through data cleaning; and training the model with the cleaned data.

[0078] In order to further implement the above technical solution, S1 specifically includes:

[0079] S11: Set the evaluation steps and generate evaluation process prompts. For example, if the evaluation steps are set to include observing the appearance and smelling the odor, the time for observing the appearance and smelling the odor are both 20 seconds. Prompt through voice broadcast: a countdown prompt is given before the start. When entering the first stage, "Observe the appearance" is played and the countdown begins. After 20 seconds, the second stage is entered and "Smell the odor" is played.

[0080] S12: Adjust the camera so that the camera's acquisition area covers the subject's face; after setting the parameters of the evaluation step, adjust the camera so that the subject's face is facing the camera, the face is 40 to 80 cm away from the camera, and the face occupies about 1 / 3 to 2 / 3 of the lens screen.

[0081] S13: Complete each tasting step according to the prompts of the tasting process, and collect corresponding video data of each tasting step through the camera.

[0082] S14: Rate your preference for the appearance and aroma of the samples. The rating questionnaire uses a [1-100] point system.

[0083] The present invention uses micro-expression analysis to capture subtle changes in facial micro-expressions of individuals when experiencing fragrances. These changes are often unconscious signals that reflect the true emotional state. Through quantitative analysis of these facial micro-expressions, objective and objective evaluation indicators can be obtained, avoiding the subjectivity of traditional subjective questionnaires. Facial micro-expressions are instantaneous and immediate reactions, so the emotional changes of individuals when experiencing fragrances can be monitored in real time. Compared with traditional subjective questionnaires that need to be filled out after the experience is over, facial micro-expression analysis technology can provide more real-time emotional experience information. Facial micro-expression analysis technology is not directly affected by environmental conditions like some traditional sensory evaluation instruments and physiological response detection methods. Compared with other sensor devices, it is relatively less affected by the environment and is easier to maintain consistency in actual use.

[0084] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0085] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fragrance preference detection system based on facial micro-expressions, characterized in that: It includes a data acquisition module and a detection module; The data acquisition module is used to create a test scene and collect the facial micro-expressions of the subjects during the test to obtain micro-expression data; The detection module is used to construct a feature vector based on the micro-expression data; when there are multiple evaluation steps in the test process, the feature vector is constructed respectively according to each evaluation step; and is used to predict the feature vector through a trained neural network model to obtain a detection result.

2. The fragrance preference detection system based on facial micro-expressions according to claim 1, characterized in that: The data acquisition module includes an information entry submodule, a parameter configuration submodule, an image acquisition submodule and a voice broadcast submodule; The information entry submodule is used to enter basic information of the subject; The parameter configuration module is used to configure the first parameter for the image acquisition submodule; and configure the second parameter for the voice broadcast submodule; The image acquisition submodule is used to confirm the position and time of image acquisition according to the first parameter; The voice broadcast submodule is used to confirm the content and timing of the voice broadcast according to the second parameter.

3. The fragrance preference detection system based on facial micro-expressions according to claim 2, characterized in that: The data collection submodule also includes a visualization operation submodule, and the visualization operation submodule is used to present the questionnaire and generate the survey results.

4. The fragrance preference detection system based on facial micro-expressions according to claim 1, characterized in that: The feature vector includes valence level, emotion concentration, and intensity of multiple emotions.

5. A fragrance preference detection system based on facial micro-expressions according to claim 1 or 4, characterized in that: The data acquisition module performs multiple samplings in each evaluation step, and the detection module performs recognition based on the micro-expression image data of the multiple samples to obtain multiple expression dimensions, and obtains the feature vector of the corresponding evaluation step by merging the recognition results of each sampling.

6. A method for detecting fragrance preference based on facial micro-expressions, characterized in that: The following steps are involved: Get facial videos during tasting; Preprocessing the facial video to construct a data set; The pre-processing step comprises: Sampling facial video samples, extracting facial movements and identifying micro-expressions, after multiple sampling and identification, multiple expression dimensions carrying micro-expression information are obtained accordingly; the micro-expression information of multiple expression dimensions is attribute-merged to confirm the feature values ​​of each expression dimension, and the feature vector corresponding to a single video sample is obtained; The detection model is trained by constructing a data set, and the trained detection model is used to detect the preference.

7. The method for detecting fragrance preference based on facial micro-expressions according to claim 6, characterized in that: The steps also include: performing data cleaning on the data set to remove outlier data, missing data and duplicate data; and training a model using the cleaned data.

8. The method for detecting fragrance preference based on facial micro-expressions according to claim 7, characterized in that: The data cleaning steps include: using the interquartile range method to observe and judge, and removing the values ​​of each input variable sequence and output observation value sequence that are less than Q L -1.5IQR or greater than Q U Values ​​of -1.5 IQR were considered outliers and were removed.

9. The method for detecting fragrance preference based on facial micro-expressions according to claim 6, characterized in that: The steps of obtaining the facial video during the evaluation include: Set the evaluation steps and generate evaluation process prompts; Adjusting the camera so that the acquisition area of ​​the camera covers the face of the subject; Complete each tasting step according to the prompts of the tasting process, and collect corresponding video data of each tasting step through the camera.

10. The method for detecting fragrance preference based on facial micro-expressions according to claim 6, characterized in that: The micro-expression information includes the intensity, valence level and expression concentration of basic emotions.

Images