Smiling state recognition method, device, equipment and medium based on multimodal features

Abstract

The present application relates to a method, device, equipment and medium for identifying a smile state based on multimodal features, wherein the method comprises: collecting multi-feature information of a subject at a preset stage, wherein the preset stage comprises a baseline stage before the subject is in a target odor environment, a stimulation stage at the time, and a recovery stage thereafter, and the multi-feature information comprises facial images, electroencephalogram signals, human posture information and sound signals; then, according to the multi-feature information of different stages, facial key point recognition, nasolabial fold area recognition, electroencephalogram signal feature recognition, human posture key point recognition and sound feature recognition are sequentially performed to obtain scores for each smile state, and then feature fusion is performed according to each smile state score to obtain a target smile state score. The present application can accurately identify whether the subject is in a smiling state at different stages in the target odor environment, thereby accurately evaluating the use effect of the product corresponding to the target odor.

Description

Technical Field

[0001] The present application relates to the technical field of emotion recognition, and in particular to a method, device, equipment and medium for identifying a smiling state based on multimodal features. Background Art

[0002] In recent years, environmental psychology research has shown that certain smells can stimulate human emotional responses, such as pleasure, relaxation, or excitement. Based on this finding, some researchers have begun to explore the use of smell as an auxiliary means of emotion recognition. In particular, certain smells have been shown to induce smiling expressions or enhance emotional states, and these smells are called "smile smells." However, there is currently no research on how to accurately identify an individual's smiling state in a specific odor environment.

[0003] Although existing technologies can recognize the emotional state of an individual to a certain extent, there are still some deficiencies in recognizing the smiling state in a specific odor environment:

[0004] Limitations of single-modality recognition: Traditional methods mainly rely on visual or auditory information, ignoring the potential impact of odor on emotional state, resulting in limited recognition accuracy in specific odor environments. Environmental interference and individual differences: The complexity of the odor environment, individual differences and the diversity of emotional expressions increase the difficulty of recognition, making it difficult for existing methods to accurately identify the smiling state. Lack of comprehensive recognition technology: There is currently a lack of a comprehensive recognition technology that can comprehensively consider odor, facial expressions and sound characteristics to improve the accuracy and reliability of smiling state recognition. And because it is difficult for existing technologies to accurately identify the smiling state at different stages of a specific odor environment, it is impossible to accurately evaluate the use effect of the product that emits the specific odor. Therefore, there is an urgent need for a smile recognition method that can accurately identify whether the subject is in a smiling state in different stages of the target odor environment, so as to accurately evaluate the use effect of the product corresponding to the target odor. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a smile state recognition method, device, equipment and medium based on multimodal features, so as to accurately identify whether the subject is in a smiling state in different stages of the target odor environment, thereby accurately evaluating the use effect of the product corresponding to the target odor.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a smile state recognition method based on multimodal features, including:

[0007] Collecting multi-feature information of the subject at a preset stage, wherein the preset stage includes a baseline stage before the subject is in the target odor environment, a stimulation stage during the stimulation stage, and a recovery stage thereafter, and the multi-feature information includes facial images, electroencephalogram signals, human posture information, and sound signals;

[0008] Respectively performing facial key point recognition on the facial images at different stages to obtain facial key point recognition results, and performing smile state recognition based on the facial key point recognition results to obtain a first smile state score for each stage;

[0009] Extracting the coordinates of the feature points of the face area in the facial image at different stages respectively to extract the image blocks of the nasolabial fold area, and performing smile state recognition according to the image blocks of the nasolabial fold area to obtain the second smile state score of each stage;

[0010] Performing feature processing on the electroencephalogram signals at different stages respectively to obtain electroencephalogram features, and performing smile state recognition according to the electroencephalogram features to obtain a third smile state score for each stage;

[0011] Respectively extracting human body posture key points from the human body posture information at different stages, determining a target body posture according to the human body posture key points, and performing smile state recognition according to the target body posture to obtain a fourth smile state score for each stage;

[0012] Extracting sound features from the sound signals at different stages respectively, and performing smile state recognition according to the sound features to obtain a fifth smile state score at each stage;

[0013] A target smile state score for each stage is determined according to the first smile state score, the second smile state score, the third smile state score, the fourth smile state score and the fifth smile state score, and a difference between the target smile state scores of any two stages is calculated to obtain a target difference.

[0014] In order to solve the above technical problems, the embodiment of the present application provides a smile state recognition device based on multimodal features, including:

[0015] A feature information acquisition unit, used to collect multi-feature information of the subject in a preset stage, wherein the preset stage includes a baseline stage before the subject is in the target odor environment, a stimulation stage during the period, and a recovery stage thereafter, and the multi-feature information includes facial images, electroencephalogram signals, human posture information, and sound signals;

[0016] a first smile state scoring unit, configured to perform face key point recognition on the facial images at different stages to obtain face key point recognition results, and perform smile state recognition based on the face key point recognition results to obtain a first smile state score for each stage;

[0017] A second smile state scoring unit is used to extract the coordinates of the feature points of the face area in the facial image at different stages to extract the nasolabial fold area image blocks, and perform smile state recognition according to the nasolabial fold area image blocks to obtain a second smile state score for each stage;

[0018] a third smile state scoring unit, configured to perform feature processing on the electroencephalogram signals at different stages to obtain electroencephalogram features, and perform smile state recognition according to the electroencephalogram features to obtain a third smile state score for each stage;

[0019] a fourth smile state scoring unit, configured to extract human body posture key points from the human body posture information at different stages, determine a target body posture according to the human body posture key points, perform smile state recognition according to the target body posture, and obtain a fourth smile state score for each stage;

[0020] a fifth smile state scoring unit, configured to extract sound features from the sound signal at different stages, and perform smile state recognition according to the sound features to obtain a fifth smile state score for each stage;

[0021] a target smile state scoring unit, for determining a target smile state score for each stage according to the first smile state score, the second smile state score, the third smile state score, the fourth smile state score and the fifth smile state score, and calculating a difference between the target smile state scores of any two stages to obtain a target difference.

[0022] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement any one of the above-mentioned smile state recognition methods based on multimodal features.

[0023] In order to solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned smile state recognition methods based on multimodal features is implemented.

[0024] The embodiment of the present invention provides a method, device, equipment and medium for smile state recognition based on multimodal features. The method includes: collecting multi-feature information of the subject at a preset stage, wherein the preset stage includes a baseline stage before the subject is in the target odor environment, a stimulation stage at the time and a recovery stage thereafter, and the multi-feature information includes facial images, electroencephalogram signals, human posture information and sound signals; performing facial key point recognition on the facial images at different stages to obtain facial key point recognition results, and performing smile state recognition based on the facial key point recognition results to obtain a first smile state score for each stage; extracting the coordinates of the feature points of the facial area in the facial images at different stages to extract the image blocks of the nasolabial fold area, and performing smile state recognition based on the image blocks of the nasolabial fold area to obtain a second smile state score for each stage; performing feature processing on the electroencephalogram signals at different stages to obtain The method comprises the steps of: extracting the human body posture key points in the human body posture information of different stages, determining the target body posture according to the human body posture key points, and performing smile state recognition according to the target body posture to obtain a fourth smile state score in each stage; extracting the sound features in the sound signals of different stages, and performing smile state recognition according to the sound features to obtain a fifth smile state score in each stage; determining the target smile state score in each stage according to the first smile state score, the second smile state score, the third smile state score, the fourth smile state score and the fifth smile state score, and calculating the difference between the target smile state scores of any two stages to obtain a target difference. The embodiment of the present invention collects multi-feature information of the subject at a preset stage, and then performs facial key point recognition, nasolabial fold area recognition, electroencephalogram signal feature recognition, human posture key point recognition and sound feature recognition in sequence according to the multi-feature information at different stages to obtain each smile state score, and then performs feature fusion according to each smile state score to obtain a target smile state score, and calculates the difference between the target smile state scores at any two stages to obtain a target difference, thereby achieving accurate identification of whether the subject is in a smiling state in different stages of the target odor environment, thereby accurately evaluating the use effect of the product corresponding to the target odor. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 It is a flow chart of the implementation of the smile state recognition method based on multimodal features provided in an embodiment of the present application;

[0027] Figure 2 is a flowchart for implementing the first sub-process in the smile state recognition method based on multimodal features provided in an embodiment of the present application;

[0028] Figure 3 is a flowchart for implementing the second sub-process in the smile state recognition method based on multimodal features provided in an embodiment of the present application;

[0029] Figure 4 is a flowchart for implementing the third sub-process in the smile state recognition method based on multimodal features provided in an embodiment of the present application;

[0030] Figure 5 is a flowchart for implementing the fourth sub-process in the smile state recognition method based on multimodal features provided in an embodiment of the present application;

[0031] Figure 6 is a flowchart for implementing the fifth sub-process in the smile state recognition method based on multimodal features provided in an embodiment of the present application;

[0032] Figure 7 is a flowchart for implementing the sixth sub-process in the smile state recognition method based on multimodal features provided in an embodiment of the present application;

[0033] Figure 8 is a schematic diagram of a smiling state recognition device based on multimodal features provided in an embodiment of the present application;

[0034] Fig. 9 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the smile state recognition method based on multimodal features provided in the embodiments of the present application is generally executed by a computer device, and accordingly, the smile state recognition device based on multimodal features is generally configured in a computer device.

[0040] See also Figure 1 , Figure 1 A specific implementation of a smiling state recognition method based on multimodal features is shown.

[0041] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:

[0042] S1: Collecting multi-feature information of the subject in a preset stage, wherein the preset stage includes a baseline stage before the subject is in the target odor environment, a stimulation stage during the period, and a recovery stage thereafter, and the multi-feature information includes facial images, electroencephalogram signals, human posture information, and sound signals.

[0043] Specifically, the multi-feature information of the subjects in the baseline phase before being in the target odor environment, the stimulation phase, and the recovery phase thereafter is obtained. Through the multi-feature information of the subjects in different stages in the target environment, the scores of the subjects in the smiling state at each stage are identified, and the smiling states at different stages are compared, so that it is possible to accurately identify whether the subjects are in a smiling state, thereby accurately evaluating the use effect of the product corresponding to the target odor. Among them, the feature information includes facial images, electroencephalogram signals, human posture information, and sound signals.

[0044] In a specific embodiment, in order to verify a certain smiling emotion product, the smiling emotion product can emit a "smile scent", that is, emit a smiling scent, which can make the subject feel smiling when smelling the scent. Therefore, in order to verify this smiling emotion product, multi-feature information of the subject when not using the product, multi-feature information when using the product environment, and multi-feature information after using the product environment are obtained. Through this application, smile score recognition is performed on the multi-feature information of different stages, the smile scores of each stage are obtained, and the difference in smile scores of each stage is evaluated, so as to evaluate the use effect of the product;

[0045] S2: performing facial key point recognition on the facial images at different stages respectively to obtain facial key point recognition results, and performing smile state recognition based on the facial key point recognition results to obtain a first smile state score for each stage.

[0046] Specifically, when a person is smiling, the shape of the mouth changes, usually the corners of the mouth rise, the eyes squint, and nasolabial folds (i.e., the lines on both sides of the nose) may appear or deepen. Therefore, the processing of the mouth and eyes can be performed by detecting the key points of the mouth and eyes, thereby performing smile recognition.

[0047] See also Figure 2 , Figure 2 A specific implementation of step S2 is shown, which is described in detail as follows:

[0048] S21: performing image preprocessing on the plurality of facial images at different stages respectively to obtain preprocessed images, and identifying face detection frames of the preprocessed images;

[0049] S22: performing facial key point recognition on the face detection frame to obtain the facial key point recognition result, and screening out target key points from the facial key point recognition result;

[0050] S23: performing smile state recognition according to the position difference of the target key points corresponding to the different facial images and the inclination angle of the connecting line of the target key points to obtain the first smile state score of each stage.

[0051] Specifically, multiple facial images at different stages are preprocessed to obtain preprocessed images. Image preprocessing includes grayscale processing, denoising, image enhancement and image cropping to a uniform specification. Then, the preprocessed image is recognized by a face recognition algorithm to obtain a face recognition frame; face key points are recognized on the face detection frame to obtain face key point recognition results, and target key points are screened out from the face key point recognition results. The target key points include left mouth key points, right mouth key points, upper jaw key points, chin key points, left eye key points, right eye key points, upper eyelid key points, lower eyelid key points, left eye key points and right eye key points. Finally, smile state recognition is performed based on the position difference of target key points corresponding to different facial images and the inclination angle of the target key point connection line to obtain the first smile state score of each stage.

[0052] See also Figure 3 , Figure 3 A specific implementation of step S23 is shown, which is described in detail as follows:

[0053] S231: Calculate the distance between the upper eyelid key point and the lower eyelid key point, the distance between the left eye key point and the right eye key point, and the distance between the upper jaw key point and the chin key point to obtain a first distance, a second distance, and a third distance respectively;

[0054] S232: Calculate the inclination angle of the connecting line between the left mouth key point and the chin key point to obtain a first inclination angle, and calculate the inclination angle of the connecting line between the right mouth key point and the chin key point to obtain a second inclination angle;

[0055] S233: Perform smile state recognition according to the position difference of the target key points corresponding to the different facial images, the first distance, the second distance, the third distance, the first tilt angle and the second tilt angle to obtain the first smile state score of each stage.

[0056] Specifically, since people often have features such as squinting eyes, upturned corners of the mouth, and slightly closed mouth when they are in a smiling state, in order to accurately obtain whether the user is smiling, the distance between the upper eyelid key point and the lower eyelid key point, the distance between the left eye key point and the right eye key point, and the distance between the upper jaw key point and the chin key point of the mouth are calculated respectively to obtain the first distance, the second distance, and the third distance. At the same time, the inclination angle of the connecting line of the left mouth key point and the chin key point is calculated to obtain the first inclination angle, and the inclination angle of the connecting line of the right mouth key point and the chin key point is calculated to obtain the second inclination angle. The smile state is recognized according to the position difference of the corresponding target key points of different facial images, the first distance, the second distance, the third distance, the first inclination angle, and the second inclination angle to obtain the first smile state score of each stage.

[0057] Furthermore, the embodiment of the present application assigns different weights to the position difference, first distance, second distance, third distance, first tilt angle and second tilt angle of different target key points, and calculates the smile score according to the weight to obtain a first smile state score. Furthermore, the facial image is used as training data, and a neural network model is trained with the training data to obtain a classification model. The classification model can perform smile recognition according to the position difference, first distance, second distance, third distance, first tilt angle and second tilt angle of the target key points, thereby obtaining a first smile state score;

[0058] S3: extracting the coordinates of the feature points of the face area in the facial image at different stages respectively to extract the image blocks of the nasolabial fold area, and performing smile state recognition based on the image blocks of the nasolabial fold area to obtain the second smile state score of each stage.

[0059] Specifically, when a person is smiling, nasolabial folds may appear on his face, so in order to further improve the detection of whether the subject is smiling, it is necessary to perform smile state recognition based on the nasolabial folds. In the embodiment of the present application, the coordinates of the feature points of the face area in the facial image at different stages are extracted respectively, and the nasolabial fold area image block is extracted according to the preset coordinates of the feature points of the face area, and then the smile state is recognized according to the nasolabial fold area image block to obtain the second smile state score of each stage.

[0060] See also Figure 4 , Figure 4 A specific implementation of step S3 is shown, which is described in detail as follows:

[0061] S31: extracting the coordinates of the feature points of the face area in the facial image at different stages respectively;

[0062] S32: confirming and extracting the nasolabial fold area image block according to the coordinates of the feature points in the face area;

[0063] S33: extracting texture features in the image block of the nasolabial fold area using a preset texture descriptor;

[0064] S34: performing smile state recognition based on the texture feature by using a preset threshold to obtain a second smile state score for each stage.

[0065] Specifically, it is necessary to extract the coordinates of feature points of the face area from the facial images at different stages. Since the nasolabial folds are located at the junction of the nose, cheeks and upper lip, it is necessary to extract the coordinates of the feature points of the nose, cheeks and upper lip from the feature point coordinates to obtain the target feature point coordinates. Then, the feature points are connected based on the target feature points, so as to determine the image block of the nasolabial fold area in the facial image, and the image block of the nasolabial fold area is cropped to obtain image blocks of different facial images, which are then cropped to a uniform size. Then, a preset texture descriptor (such as local binary pattern LBP, gray level co-occurrence matrix GLCM, etc.) is used to extract the texture features in the image block of the nasolabial fold area. These texture features can describe the local texture information of the image block, such as the thickness and direction of the texture, which is crucial for identifying the smiling state. Finally, the extracted texture features are analyzed using a preset threshold to identify the smiling state. The present application can use a classifier (such as a support vector machine SVM, a decision tree, etc.), which determines whether the face in the image is in a smiling state based on the texture features. For each stage of the image, a smile state score is obtained, thereby obtaining a second smile state score, which reflects the confidence or possibility of the smile state in the facial image at that stage;

[0066] S4: performing feature processing on the EEG signals at different stages respectively to obtain EEG features, and performing smile state recognition according to the EEG features to obtain a third smile state score for each stage.

[0067] Specifically, by analyzing the EEG signal of the subject, the smiling state of the subject at different stages can be identified, and a smiling state score is given for each stage, thereby obtaining the third smiling state score for each stage.

[0068] See also Figure 5 , Figure 5 A specific implementation of step S4 is shown, which is described in detail as follows:

[0069] S41: Preprocessing the electroencephalogram signals at different stages respectively to obtain preprocessed electroencephalogram signals.

[0070] Specifically, in order to remove noise and interference in the EEG signal and improve the signal quality, it is necessary to preprocess the EEG signal at different stages to obtain the preprocessed EEG signal. Image preprocessing includes filtering (such as bandpass filtering, high-pass filtering or low-pass filtering), removing baseline drift, removing eye movement and electromyography interference, etc.;

[0071] S42: extracting the time-frequency features of the preprocessed EEG signal within the same time length, and constructing a target sliding window.

[0072] Specifically, features that can reflect the time domain and frequency domain characteristics of the signal are extracted from the preprocessed EEG signal. These features include amplitude, frequency, phase, etc. In order to ensure that the selected time length is the same when extracting features from the EEG signal of each stage for comparison, it is necessary to extract video features within the same time. Finally, a sliding window is constructed to slide on the EEG signal and extract features. The size of this window and the sliding step can be adjusted as needed. Finally, the time-frequency features of each stage within the same time length are obtained, and the sliding window for subsequent analysis is prepared;

[0073] S43: Inputting the time-frequency features into the SVM model through the target sliding window to perform smile state recognition, and obtaining the third smile state score of each stage.

[0074] Specifically, Support Vector Machine (SVM) is a commonly used classification algorithm, which is particularly suitable for classification problems of high-dimensional data. The extracted time-frequency features are input into the SVM model through a sliding window. In the model training stage, it is necessary to use data with known smile state labels to train the SVM model so that it can learn the association between the smile state and the EEG features. In the model prediction stage, the trained SVM model is used to perform smile state recognition on the new EEG signal. According to the output of the SVM model, a smile state score is calculated for each stage to obtain the third smile state score for each stage. Specifically, the SVM model can be trained in advance, and in actual application scenarios, the smile state is recognized by using the trained SVM model to obtain the third smile state score for each stage;

[0075] S5: extracting human body posture key points from the human body posture information at different stages respectively, determining a target body posture according to the human body posture key points, performing smile state recognition according to the target body posture, and obtaining a fourth smile state score for each stage.

[0076] Specifically, the human posture information includes the information of each key part of the subject captured by the camera or sensor and the human posture image. When the subject is in a smiling state, the head may be slightly raised, or kept upright, the shoulders relaxed and the back straight, and the body leaning forward, so the key points of each part of the user's body can be extracted (posture estimation algorithms, such as OpenPose, MediaPipe Pose, etc., detect human key points (such as joints, feature points, etc.) from images or videos in real time, and infer the overall posture of the human body through the connection and relationship between key points), and judge whether the user has the above situation, and then classify according to different weights or confidence levels. During this period, data preprocessing and feature extraction can also be performed on the human posture information; then the labeled emotional data is used to train machine learning models, such as support vector machines (SVM), naive Bayes, etc. These models predict the emotional tendency of new data by learning emotional features in text. The extracted features are emotionally classified to identify body postures related to emotions, so that the classifier is classified to obtain the fourth smile state score of each stage.

[0077] See also Figure 6 , Figure 6 A specific implementation of step S5 is shown, which is described in detail as follows:

[0078] S51: performing image preprocessing on the human body posture information at different stages respectively, and extracting human body posture key points from the human body posture information after image preprocessing by using a preset posture estimation algorithm.

[0079] Specifically, the human posture images at different stages are preprocessed. Preprocessing includes adjusting image size, denoising, graying, edge detection, etc. to improve the accuracy of subsequent posture estimation. The preprocessed images are then processed by a preset posture estimation algorithm (such as a deep learning model). The algorithm can identify and extract key points of human posture, such as joints (shoulders, elbows, knees, etc.) and other important body parts;

[0080] S52: Compare the human body posture key point information with the human body posture key point to determine the target human body posture key point.

[0081] Specifically, the extracted human posture key point information is compared with the preset human posture key points. The preset key points here represent the key positions of a certain standard posture or a specific action. Then, through comparison, the key point set that best matches the target posture is determined, and these key points are the target human posture key points;

[0082] S53: determining the target body posture according to the target human body posture, and performing smile state recognition according to the target body posture to obtain a fourth smile state score of each stage.

[0083] Specifically, the target body posture of the human body can be determined based on the key points of the target body posture. The target body posture performs geometric analysis on the key points to identify whether the human body is standing, sitting, leaning forward or in some other posture. After the target body posture is determined, smile state recognition is performed. The machine learning model is trained using the labeled emotion data, and the smile state is recognized based on the target body posture through the machine learning model to obtain the fourth smile state score for each stage. The annotation in the machine learning model is to annotate each body posture of keeping upright, relaxing shoulders, straightening the back, and leaning forward, assigning different weights to these postures, training the model through the annotation information and weight information, and identifying different body posture scores according to the model, and using this score as the fourth smile state score. Since the confidence level of judging whether the subject is in a smiling state by body posture is low, the weight ratio of this feature can be reduced in subsequent feature fusion;

[0084] S6: extracting sound features from the sound signal at different stages respectively, and performing smile state recognition according to the sound features to obtain a fifth smile state score for each stage.

[0085] Specifically, when smiling early, a person's voice tends to become softer and warmer. This is because smiling relaxes facial muscles, including muscles related to vocalization, making the voice sound more natural and comfortable. At the same time, when smiling, the degree of opening of the oral cavity and nasal cavity may change, thereby affecting the resonance of the voice. This leads to subtle changes in the sound quality, pitch and volume of the voice. Therefore, it is possible to identify whether the current sound state is in a smiling state, and it is also possible to combine the changes in the user's voice before and after to comprehensively judge whether it is in a smiling state. It can use sound analysis tools (such as audio editing software, sound recognition software, etc.) to perform spectrum analysis and waveform observation on the sound, such as judging pitch, volume, speech speed, rhythm, etc., and can capture subtle changes in the sound, thereby assisting in judging whether the person is in a smiling state. By training the machine learning algorithm, the characteristic pattern of the sound when smiling can also be identified. Therefore, in the embodiment of the present application, the sound features in the sound signal at different stages are extracted respectively, and the smiling state is identified according to the sound features to obtain the fifth smiling state score of each stage.

[0086] See also Figure 7 , Figure 7 A specific implementation of step S6 is shown, which is described in detail as follows:

[0087] S61: extracting the sound features from the sound signal at different stages respectively, wherein the sound features include pitch, volume and speech speed;

[0088] S62: segmenting the sound signal, and comparing the segmented sound signals with the corresponding sound features to obtain a comparison result;

[0089] S63: Perform smile state recognition based on the comparison result to obtain the fifth smile state score of each stage.

[0090] Specifically, the continuous sound signal is divided into several segments, each segment representing the sound within a time window. Segmentation helps to analyze the changes in sound features more carefully. For the segmented sound signal, the corresponding sound features (pitch, volume, speaking speed) are compared. This comparison involves comparing the sound features of the current segment with a preset standard or the sound features of the previous segment. The purpose of the comparison is to identify changes or anomalies in the sound features, which may be related to the smiling state. Based on the comparison results of the sound features, the smiling state is identified. Based on the changes in the pitch, volume and speaking speed in the comparison results, the smiling state is identified to obtain the fifth smiling state score for each stage. When performing smiling state recognition, different weights are assigned to different pitches, volumes and speaking speeds, as well as the comparison results of the pitch, volume and speaking speed, and the smile score of the subject at this stage is calculated based on the weight information;

[0091] S7: Determine a target smile state score for each stage according to the first smile state score, the second smile state score, the third smile state score, the fourth smile state score and the fifth smile state score, and calculate the difference between the target smile state scores of any two stages to obtain a target difference.

[0092] Specifically, the above steps have obtained different smile scores according to different features, but different features have different confidence levels for smile state scores. For example, the key points, nasolabial folds and electroencephalogram signals of facial images have higher reliability in identifying smile states than human posture information and sound features, so different weights can be assigned to different smile state scores, and the target smile state score for each stage is calculated based on the weight and the smile state score. Then, in order to verify the effect of the odor product before and after use and during use, the difference between the target smile state scores of any two stages is calculated to obtain the target difference. For example, the higher the target difference between the score in the target odor environment minus the score before use, the higher the possibility that the product makes people smile, and the better the use effect of the product.

[0093] In an embodiment of the present application, multi-feature information of the subject at a preset stage is collected, wherein the preset stage includes a baseline stage before the subject is in the target odor environment, a stimulation stage during the stimulation stage, and a recovery stage thereafter, and the multi-feature information includes facial images, electroencephalogram signals, human posture information, and sound signals; facial key point recognition is performed on the facial images at different stages to obtain facial key point recognition results, and smile state recognition is performed based on the facial key point recognition results to obtain a first smile state score for each stage; the coordinates of feature points of the facial area in the facial images at different stages are extracted to extract image blocks of the nasolabial fold area, and smile state recognition is performed based on the image blocks of the nasolabial fold area to obtain a second smile state score for each stage; feature processing is performed on the electroencephalogram signals at different stages to obtain The method comprises the steps of: extracting the human body posture key points in the human body posture information of different stages, determining the target body posture according to the human body posture key points, and performing smile state recognition according to the target body posture to obtain a fourth smile state score in each stage; extracting the sound features in the sound signals of different stages, and performing smile state recognition according to the sound features to obtain a fifth smile state score in each stage; determining the target smile state score in each stage according to the first smile state score, the second smile state score, the third smile state score, the fourth smile state score and the fifth smile state score, and calculating the difference between the target smile state scores of any two stages to obtain a target difference. The embodiment of the present invention collects multi-feature information of the subject at a preset stage, and then performs facial key point recognition, nasolabial fold area recognition, electroencephalogram signal feature recognition, human posture key point recognition and sound feature recognition in sequence according to the multi-feature information at different stages to obtain each smile state score, and then performs feature fusion according to each smile state score to obtain a target smile state score, and calculates the difference between the target smile state scores at any two stages to obtain a target difference, thereby achieving accurate identification of whether the subject is in a smiling state in different stages of the target odor environment, thereby accurately evaluating the use effect of the product corresponding to the target odor.

[0094] Please refer to Figure 8 , as a response to the above Figure 1 The present application provides an embodiment of a smile state recognition device based on multimodal features, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0095] like Figure 8As shown, the smile state recognition device based on multimodal features of this embodiment includes: a feature information collection unit 81, a first smile state scoring unit 82, a second smile state scoring unit 83, a third smile state scoring unit 84, a fourth smile state scoring unit 85, a fifth smile state scoring unit 86 and a target smile state scoring unit 87, wherein:

[0096] The feature information acquisition unit 81 is used to collect multi-feature information of the subject in a preset stage, wherein the preset stage includes a baseline stage before the subject is in the target odor environment, a stimulation stage during the stimulation stage, and a recovery stage thereafter, and the multi-feature information includes facial images, electroencephalogram signals, human posture information, and sound signals;

[0097] A first smile state scoring unit 82 is used to perform face key point recognition on the facial images at different stages to obtain face key point recognition results, and perform smile state recognition based on the face key point recognition results to obtain a first smile state score for each stage;

[0098] A second smile state scoring unit 83 is used to extract the coordinates of the feature points of the face area in the facial image at different stages to extract the nasolabial fold area image blocks, and perform smile state recognition according to the nasolabial fold area image blocks to obtain the second smile state score of each stage;

[0099] A third smile state scoring unit 84 is used to perform feature processing on the EEG signals at different stages to obtain EEG features, and perform smile state recognition based on the EEG features to obtain a third smile state score for each stage;

[0100] A fourth smile state scoring unit 85 is used to extract human body posture key points in the human body posture information at different stages, determine a target body posture according to the human body posture key points, perform smile state recognition according to the target body posture, and obtain a fourth smile state score for each stage;

[0101] A fifth smile state scoring unit 86, configured to extract sound features from the sound signal at different stages, and perform smile state recognition according to the sound features to obtain a fifth smile state score for each stage;

[0102] The target smile state scoring unit 87 is used to determine the target smile state score for each stage according to the first smile state score, the second smile state score, the third smile state score, the fourth smile state score and the fifth smile state score, and calculate the difference between the target smile state scores of any two stages to obtain a target difference.

[0103] Further, the first smile state scoring unit 82 includes:

[0104] A face detection frame recognition unit, used to perform image preprocessing on the plurality of facial images at different stages to obtain preprocessed images, and recognize face detection frames of the preprocessed images;

[0105] A facial key point recognition unit, used to perform facial key point recognition on the face detection frame, obtain the facial key point recognition result, and filter out target key points from the facial key point recognition result;

[0106] The first smile recognition unit is used to perform smile state recognition according to the position difference of the target key points corresponding to the different facial images and the inclination angle of the target key point connection line to obtain the first smile state score in each stage.

[0107] Further, the target key points include left mouth key points, right mouth key points, upper jaw key points, chin key points, left eye key points, right eye key points, upper eyelid key points, lower eyelid key points, left eye key points and right eye key points; the first smile recognition unit includes:

[0108] A key point distance calculation unit, used to respectively calculate the distance between the upper eyelid key point and the lower eyelid key point, the distance between the left eye key point and the right eye key point, and the distance between the upper jaw key point and the chin key point, to obtain a first distance, a second distance, and a third distance;

[0109] an inclination angle calculation unit, used to calculate the inclination angle of the connecting line of the left mouth key point and the chin key point to obtain a first inclination angle, and to calculate the inclination angle of the connecting line of the right mouth key point and the chin key point to obtain a second inclination angle;

[0110] The first smile score calculation unit is used to perform smile state recognition according to the position difference of the target key points corresponding to the different facial images, the first distance, the second distance, the third distance, the first tilt angle and the second tilt angle to obtain the first smile state score for each stage.

[0111] Further, the second smile state scoring unit 83 includes:

[0112] A feature point coordinate extraction unit, used to extract the coordinates of the feature points of the face area in the facial image at different stages respectively;

[0113] An image block confirmation unit, used to confirm and extract the nasolabial fold area image block according to the coordinates of the feature points in the face area;

[0114] A texture feature extraction unit, used to extract texture features in the image block of the nasolabial fold area using a preset texture descriptor;

[0115] The second smile recognition unit is used to perform smile state recognition based on the texture feature through a preset threshold to obtain a second smile state score for each stage.

[0116] Further, the third smile state scoring unit 84 includes:

[0117] An electroencephalogram signal preprocessing unit, used to preprocess the electroencephalogram signals at different stages to obtain preprocessed electroencephalogram signals;

[0118] A time-frequency feature extraction unit, used to extract the time-frequency features of the preprocessed EEG signal within the same time length and construct a target sliding window;

[0119] The third smile recognition unit is used to input the time-frequency feature into the SVM model through the target sliding window to perform smile state recognition, so as to obtain the third smile state score of each stage.

[0120] Furthermore, the human body posture information includes information of various key parts of the subject and a human body posture image captured by a camera or a sensor; the fourth smile state scoring unit 85 includes:

[0121] A human body posture key point extraction unit is used to perform image preprocessing on the human body posture information at different stages, and extract human body posture key points from the human body posture information after image preprocessing by a preset posture estimation algorithm;

[0122] A human body posture key point confirmation unit, used for comparing the human body posture key point information with the human body posture key point to determine the target human body posture key point;

[0123] The fourth smile recognition unit is used to determine the target body posture according to the target human body posture, and perform smile state recognition according to the target body posture to obtain a fourth smile state score for each stage.

[0124] Further, the fifth smile state scoring unit 86 includes:

[0125] A sound feature extraction unit, used to extract the sound features in the sound signal at different stages, wherein the sound features include pitch, volume and speech speed;

[0126] A sound feature comparison unit, used to segment the sound signal, and compare the segmented sound signals with the corresponding sound features to obtain a comparison result;

[0127] The fifth smile recognition unit is used to perform smile state recognition based on the comparison result to obtain the fifth smile state score of each stage.

[0128] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0129] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 which are interconnected and communicated via a system bus. It should be noted that: Fig. 9 Only a computer device 9 having three components, memory 91, processor 92, and network interface 93, is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0130] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0131] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 can also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 9. Of course, the memory 91 can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as the program code of the smile state recognition method based on multimodal features, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0132] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run the program code stored in the memory 91 or process data, such as running the program code of the above-mentioned smile state recognition method based on multimodal features to implement various embodiments of the smile state recognition method based on multimodal features.

[0133] The network interface 93 may include a wireless network interface or a wired network interface, and the network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0134] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores a computer program. The computer program can be executed by at least one processor to enable the at least one processor to perform the steps of a smile state recognition method based on multimodal features as described above.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0136] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is similarly within the scope of protection of the present application.

Claims

1. A smile state recognition method based on multimodal features, characterized in that: include: Collecting multi-feature information of the subject at a preset stage, wherein the preset stage includes a baseline stage before the subject is in the target odor environment, a stimulation stage during the stimulation stage, and a recovery stage thereafter, and the multi-feature information includes facial images, electroencephalogram signals, human posture information, and sound signals; Respectively performing facial key point recognition on the facial images at different stages to obtain facial key point recognition results, and performing smile state recognition based on the facial key point recognition results to obtain a first smile state score for each stage; Extracting the coordinates of the feature points of the face area in the facial image at different stages respectively to extract the image blocks of the nasolabial fold area, and performing smile state recognition according to the image blocks of the nasolabial fold area to obtain the second smile state score of each stage; Performing feature processing on the electroencephalogram signals at different stages respectively to obtain electroencephalogram features, and performing smile state recognition according to the electroencephalogram features to obtain a third smile state score for each stage; Respectively extracting human body posture key points from the human body posture information at different stages, determining a target body posture according to the human body posture key points, and performing smile state recognition according to the target body posture to obtain a fourth smile state score for each stage; Extracting sound features from the sound signals at different stages respectively, and performing smile state recognition according to the sound features to obtain a fifth smile state score at each stage; Determining a target smile state score for each stage according to the first smile state score, the second smile state score, the third smile state score, the fourth smile state score and the fifth smile state score, and calculating the difference between the target smile state scores of any two stages to obtain a target difference, and using the target difference as an indicator for evaluating the effect of a target scent product; The facial key point recognition is performed on the facial images at different stages to obtain facial key point recognition results, and the smile state recognition is performed according to the facial key point recognition results to obtain the first smile state score of each stage, including: Performing image preprocessing on the plurality of facial images at different stages respectively to obtain preprocessed images, and identifying face detection frames of the preprocessed images; Performing facial key point recognition on the face detection frame to obtain the facial key point recognition result, and screening out target key points from the facial key point recognition result, wherein the target key points include left mouth key points, right mouth key points, upper jaw key points, chin key points, left eye key points, right eye key points, upper eyelid key points, lower eyelid key points, left eye key points and right eye key points; Calculate the distance between the upper eyelid key point and the lower eyelid key point, the distance between the left eye key point and the right eye key point, and the distance between the upper jaw key point and the chin key point to obtain a first distance, a second distance, and a third distance respectively; Calculating the inclination angle of the connecting line of the left mouth key point and the chin key point to obtain a first inclination angle, and calculating the inclination angle of the connecting line of the right mouth key point and the chin key point to obtain a second inclination angle; Different weights are assigned to the position differences of the target key points corresponding to the different facial images, the first distance, the second distance, the third distance, the first tilt angle and the second tilt angle, and smile state recognition is performed according to the assigned weights to obtain the first smile state score for each stage.

2. The smile state recognition method based on multimodal features according to claim 1, characterized in that: The extracting the coordinates of the feature points of the face area in the facial image at different stages to extract the nasolabial fold area image block, and performing smile state recognition according to the nasolabial fold area image block to obtain the second smile state score of each stage includes: Respectively extracting the coordinates of the feature points of the face area in the facial images at different stages; Confirm and extract the nasolabial fold area image block according to the coordinates of the feature points in the face area; Using a preset texture descriptor to extract texture features in the image block of the nasolabial fold area; The smile state is identified based on the texture feature using a preset threshold to obtain a second smile state score for each stage.

3. The smile state recognition method based on multimodal features according to claim 1, characterized in that: The step of performing feature processing on the electroencephalogram signals at different stages to obtain electroencephalogram features, and performing smile state recognition according to the electroencephalogram features to obtain a third smile state score for each stage includes: Preprocessing the electroencephalogram signals at different stages respectively to obtain preprocessed electroencephalogram signals; Extracting the time-frequency features of the preprocessed EEG signal within the same time period, and constructing a target sliding window; The time-frequency features are input into the SVM model through the target sliding window to perform smile state recognition, so as to obtain the third smile state score of each stage.

4. The smile state recognition method based on multimodal features according to any one of claims 1 to 3, characterized in that: The human body posture information includes information of various key parts of the subject and a human body posture image captured by a camera or a sensor; extracting human body posture key points in the human body posture information at different stages, determining a target body posture according to the human body posture key points, performing smile state recognition according to the target body posture, and obtaining a fourth smile state score for each stage, including: Performing image preprocessing on the human body posture information at different stages respectively, and extracting human body posture key points from the human body posture information after image preprocessing by using a preset posture estimation algorithm; Comparing the human body posture key point information with the human body posture key point to determine the target human body posture key point; The target body posture is determined according to the target human body posture, and the smile state is recognized according to the target body posture to obtain a fourth smile state score of each stage.

5. The smile state recognition method based on multimodal features according to any one of claims 1 to 3, characterized in that: The extracting the sound features in the sound signal at different stages respectively, and performing smile state recognition according to the sound features to obtain the fifth smile state score of each stage, comprises: Respectively extracting the sound features from the sound signal at different stages, wherein the sound features include pitch, volume and speech speed; Segmenting the sound signal, and comparing the segmented sound signals with the corresponding sound features to obtain a comparison result; Smile state recognition is performed based on the comparison result to obtain the fifth smile state score of each stage.

6. A smile state recognition device based on multimodal features, characterized in that: include: A feature information acquisition unit, used to collect multi-feature information of the subject in a preset stage, wherein the preset stage includes a baseline stage before the subject is in the target odor environment, a stimulation stage during the period, and a recovery stage thereafter, and the multi-feature information includes facial images, electroencephalogram signals, human posture information, and sound signals; a first smile state scoring unit, configured to perform face key point recognition on the facial images at different stages to obtain face key point recognition results, and perform smile state recognition based on the face key point recognition results to obtain a first smile state score for each stage; A second smile state scoring unit is used to extract the coordinates of the feature points of the face area in the facial image at different stages to extract the nasolabial fold area image blocks, and perform smile state recognition according to the nasolabial fold area image blocks to obtain a second smile state score for each stage; a third smile state scoring unit, configured to perform feature processing on the electroencephalogram signals at different stages to obtain electroencephalogram features, and perform smile state recognition according to the electroencephalogram features to obtain a third smile state score for each stage; a fourth smile state scoring unit, configured to extract human body posture key points from the human body posture information at different stages, determine a target body posture according to the human body posture key points, perform smile state recognition according to the target body posture, and obtain a fourth smile state score for each stage; a fifth smile state scoring unit, configured to extract sound features from the sound signal at different stages, and perform smile state recognition according to the sound features to obtain a fifth smile state score for each stage; a target smile state scoring unit, configured to determine a target smile state score for each stage according to the first smile state score, the second smile state score, the third smile state score, the fourth smile state score and the fifth smile state score, and calculate a difference between the target smile state scores of any two stages to obtain a target difference, and use the target difference as an indicator for evaluating the effect of a target scent product; The first smile state scoring unit comprises: A face detection frame recognition unit, used to perform image preprocessing on the plurality of facial images at different stages to obtain preprocessed images, and recognize face detection frames of the preprocessed images; A face key point recognition unit is used to perform face key point recognition on the face detection frame to obtain the face key point recognition result, and filter out target key points from the face key point recognition result, wherein the target key points include left mouth key points, right mouth key points, upper jaw key points, chin key points, left eye key points, right eye key points, upper eyelid key points, lower eyelid key points, left eye key points and right eye key points; A key point distance calculation unit, used to respectively calculate the distance between the upper eyelid key point and the lower eyelid key point, the distance between the left eye key point and the right eye key point, and the distance between the upper jaw key point and the chin key point, to obtain a first distance, a second distance, and a third distance; an inclination angle calculation unit, used to calculate the inclination angle of the connecting line of the left mouth key point and the chin key point to obtain a first inclination angle, and to calculate the inclination angle of the connecting line of the right mouth key point and the chin key point to obtain a second inclination angle; The first smile score calculation unit is used to assign different weights to the position difference of the target key points corresponding to different facial images, the first distance, the second distance, the third distance, the first tilt angle and the second tilt angle, and perform smile state recognition according to the assigned weights to obtain the first smile state score for each stage.

7. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the smile state recognition method based on multimodal features as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying a smiling state based on multimodal features according to any one of claims 1 to 5 is implemented.

Images