Biological emotion recognition device
By combining data collection, feature extraction, and machine learning models, accurate identification and understanding of pet emotions have been achieved, solving the problem of inaccurate pet emotion identification in existing technologies and improving the quality of interaction between pet owners and their pets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2026-03-10
AI Technical Summary
Existing emotion recognition technologies struggle to accurately identify and analyze pets' emotional states through multimodal data fusion and detailed emotion classification, thus impacting the relationship between owners and pets.
The system uses a data acquisition module to obtain facial expression, behavioral, physiological, and sound wave data. It then uses convolutional neural networks, hidden Markov models, Fourier transforms, and Mel frequency cepstral coefficients for feature extraction. Finally, it combines these with a machine learning classification model for training and prediction, and displays the emotion prediction results in the display module.
It enables accurate identification and understanding of pets' emotions, helping owners adjust their approach and improve the harmony between owners and their pets.
Smart Images
Figure CN121640535A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to a biological emotion recognition device, particularly comprising a data acquisition module, a data processing module, a feature fusion module, a model training and prediction module, and a display module. Various sensing devices are used to obtain various data, which are analyzed and calculated to obtain accurate emotional responses of pets by pet owners, so that the pet owners can better understand the pets and adjust their treatment methods to achieve a more harmonious relationship between the pet owners and the pets. BACKGROUND
[0002] Emotion recognition technology is an innovative technology that uses various methods and tools to interpret and analyze biological emotional states, particularly for pets (such as cats, dogs, or other animals) that are in close contact with humans. The core of this technology is to integrate and analyze data from different sources to accurately identify and understand emotional responses.
[0003] However, emotion recognition technology can accurately identify and analyze the emotional state of animals through multi-modal data fusion and detailed emotion classification, which has broad potential in medical applications and breeding prediction. With the advancement of technology, emotion recognition technology will play an increasingly important role in animal welfare and behavior research. How to obtain various data through sensing devices, analyze and calculate to obtain accurate emotional responses of pets by pet owners, so that the pet owners can better understand the pets and adjust their treatment methods is the goal and problem that the present application aims to achieve. SUMMARY
[0004] Therefore, in view of the above problems and deficiencies, the present application aims to provide a biological emotion recognition device.
[0005] The present application provides a biological emotion recognition device, comprising: a data acquisition module, which obtains biological expression data and behavior data through a camera; physiological data through a contact or non-contact sensor; and sound wave data through a microphone; a data processing module, which extracts expression features, behavior features, physiological features, and sound wave features based on expression data, behavior data, physiological data, and sound wave data; a feature fusion module, which processes and calculates expression features, behavior features, physiological features, and sound wave features to form a comprehensive feature vector; a model training and prediction module, which trains a model through a machine learning classification module based on the comprehensive feature vector, and performs emotion prediction on real-time data of the trained model to obtain emotion prediction data; and a display module, which displays the emotion prediction data or converts the emotion prediction data into emotion prediction text, emotion prediction graphics, or emotion prediction animation through a conversion module and displays them.
[0006] Through the above data acquisition module, data processing module, feature fusion module, model training and prediction module and display module, various data obtained by various sensing devices are analyzed and calculated to make the pet owner obtain the accurate emotional response of the pet, so as to better understand the pet and adjust the treatment method, so as to achieve the purpose of making the pet owner and the pet more harmonious.
[0007] The data processing module of the present application obtains the expression data, and performs image processing and expression data feature extraction through a convolutional neural network (CNN), and the feature mapping F of the expression data feature extraction is calculated as follows:
[0008] F = ReLU (W * I + b) ;
[0009] Where W is a convolution kernel, I is an input image, b is a bias, and ReLU is an activation function.
[0010] The data processing module of the present application obtains the behavior data, and uses a hidden Markov model (HMM) or a long short-term memory network (LSTM) time series data analysis method to perform behavior feature extraction, and the probability calculation formula of the hidden Markov model is as follows:
[0011]
[0012] Where O is the observed sequence, Q is the state sequence, and lambda is the model parameter.
[0013] The data processing module of the present application obtains the physiological data, and uses Fourier transform to analyze heart rate variability (HRV) and respiratory rate, and the calculation formula of the Fourier transform is as follows:
[0014]
[0015] Where f(x) is the signal in the time domain, and F(t) is the representation in the frequency domain.
[0016] The data processing module of the present application obtains the sound wave data, and uses the Mel frequency cepstral coefficient (MFCC) to perform sound wave feature extraction, and the calculation formula of the Mel frequency cepstral coefficient is as follows:
[0017]
[0018] Where S(m) is the power spectrum, M is the number of filters, and n is the index of the coefficient.
[0019] The model training and prediction module of the present application is trained by the machine learning classification module, which uses support vector machine (SVM), random forest or deep learning model (such as multi-layer perceptron (MLP)) for classification, and the optimization calculation formula of the support vector machine is as follows:
[0020]
[0021] Subject: y i (w·x i +b)≥1-ξ i
[0022] Where ω is the weight vector; x i is the input feature vector; y i is the label; ξ i is the slack variable; and C is the regularization parameter. BRIEF DESCRIPTION OF DRAWINGS
[0023] Figure 1 is a functional block diagram of the biological emotion recognition device of the present application.
[0024] Figure 2 is a method flowchart for feature fusion performed by the feature fusion module of the present application.
[0025] Figure 3 is a physiological state example schematic diagram of an embodiment of the present application.
[0026] Figure 4 is a sound example schematic diagram of an embodiment of the present application.
[0027] BRIEF DESCRIPTION OF DRAWINGS: 100 - electronic device; 1 - biological; 2 - data acquisition module; 21 - camera; 22 - contact or non-contact sensor; 23 - microphone; 3 - data processing module; 4 - feature fusion module; 5 - model training and prediction module; 51 - conversion module; 52 - intelligent feedback module; 6 - display module; Step A1 - aligning the expression features, behavior features, physiological features and sound wave features in the time scale; Step A2 - splicing the expression features, the behavior features, the physiological features and the sound wave features into the comprehensive feature vector; Step A3 - using dimension reduction technology to reduce the dimension of the comprehensive feature vector, which can reduce redundant information and improve computational efficiency. DETAILED DESCRIPTION
[0028] In order to achieve the above-mentioned purposes and effects, the technical means and structure adopted by the present application are described in detail as follows with reference to the preferred embodiments of the present application, so as to fully understand the features and functions.
[0029] Please refer to Figure 1As shown, it is a function block diagram of the biological emotion recognition device of the present application, from which it can be clearly seen that the biological emotion recognition device mainly comprises a data acquisition module 2, a data processing module 3, a feature fusion module 4, a model training and prediction module 5 and a display module 6, and the data acquisition module 2, the data processing module 3, the feature fusion module 4, the model training and prediction module 5 and the display module 6 are arranged in an electronic device 100, which can be a personal computer (PC), a notebook computer (NoteBook), a tablet computer (Table PC) or a smart phone (Smart-phone), and the data acquisition module 2, the data processing module 3, the feature fusion module 4 and the model training and prediction module 5 are stored in a storage unit (such as a hard disk (HDD) or a solid state disk (SSD)) of the electronic device 100 and can be an application program (Application), and the main components and features of the biological emotion recognition device are described in detail as follows:
[0030] The data acquisition module 2 captures the facial image and video of the biological 1 (such as a cat, a dog or other animals) through a camera 21 to obtain expression data, and the camera 21 also captures the behavior action of the biological 1 to obtain behavior data; collects the heart rate and breathing rate of the biological 1 through a contact or non-contact sensor 22 to obtain physiological data; and records the sound of the biological 1 through a microphone 23 to obtain sound wave data.
[0031] The data processing module 3 obtains the expression data, the behavior data, the physiological data and the sound wave data from the data acquisition module 2 and processes them to extract expression features, behavior features, physiological features and sound wave features.
[0032] The feature fusion module 4 obtains the expression features, the behavior features, the physiological features and the sound wave features from the data processing module 3 and processes them to form a comprehensive feature vector.
[0033] The model training and prediction module 5 obtains the comprehensive feature vector from the feature fusion module 4, trains a model through a machine learning classification module, and predicts emotions for real-time data of the trained model to obtain an emotion prediction data.
[0034] The display module 6 obtains the emotion prediction data from the model training and prediction module 5 and displays it, or converts the emotion prediction data into emotion prediction text, emotion prediction graphics or emotion prediction animation through a conversion module 51 and displays it.
[0035] The contact sensor of the data acquisition module 2 for obtaining physiological data is a wearable sensor, and the non-contact sensor is an optical sensor.
[0036] The detailed execution function of the data collection module 2 is as follows:
[0037] 1. Data collection and preprocessing:
[0038] 1-1 Data collection: Collect real-time physiological, image, and sound data of pets through multi-modal sensors. Collect facial images of cats and dogs in different emotional states. For example:
[0039] Happy (joy), angry, scared, calm, tired.
[0040] 1-2 Data cleaning: Filter out noise and invalid data to ensure data quality.
[0041] 1-3 Data standardization: Convert data to a unified format for subsequent processing.
[0042] 2. Data labeling:
[0043] 2-1 Manual labeling: Label the emotional state of pets according to observation by experts, and generate a labeled data set.
[0044] 2-2 Automatic labeling: Use pre-trained models to automatically label a large amount of unlabeled data and generate a preliminary labeled data set.
[0045] 2-3 Key point labeling:
[0046] Use pre-trained cat and dog labeling models (such as Dlib) to label key points on the faces of cats and dogs. These key points may include ear, eye, nose, mouth, etc.
[0047] Labeling conditions:
[0048] 1. Use pre-trained cat and dog labeling models.
[0049] 2. Label specific points on the face, such as 68 or more key points.
[0050] 3. Ensure the accuracy of the labeled points under different facial expressions.
[0051] Examples:
[0052] Eyes: center of left eye, center of right eye.
[0053] Ears: left ear tip, right ear tip.
[0054] Nose: nose tip.
[0055] Mouth: left corner of mouth, right corner of mouth.
[0056] After the data processing module 3 obtains the expression data, it performs image processing and expression data feature extraction through a convolutional neural network (CNN). The feature mapping F of the expression data feature extraction is calculated as follows:
[0057] F = ReLU (W • I + b);
[0058] where W is the convolution kernel, I is the input image, b is the bias, and ReLU is the activation function.
[0059] After the data processing module 3 obtains the behavior data, it uses a hidden Markov model (HMM) or a long short-term memory network (LSTM) time series data analysis method to perform behavior feature extraction. The probability calculation formula of the hidden Markov model is as follows:
[0060]
[0061] where O is the observed sequence, Q is the state sequence, and λ is the model parameter.
[0062] Furthermore, expression feature extraction also includes extracting feature vectors from labeled facial key points. These feature vectors can be based on geometric features (such as the distance and angle between key points) or deep learning-based feature representations.
[0063] Geometric feature calculation method:
[0064] 1. Euclidean distance:
[0065] Calculate the Euclidean distance between two key points.
[0066] Distance formula:
[0067]
[0068] Example:
[0069] Distance from left ear tip (x1, y1) to right ear tip (x2, y2)
[0070] 2. Angle:
[0071] Calculate the angle formed by three key points.
[0072] Angle formula:
[0073]
[0074] Deep learning features, such as the following steps:
[0075] 1. Use a convolutional neural network (CNN):
[0076] 2. Input the biological (cat or dog) face image into a pre-trained CNN model (e.g. ResNet).
[0077] 3. Extract the intermediate layer features of the CNN to generate a high-dimensional vector (e.g. 128-dimensional or 512-dimensional).
[0078] After the data processing module obtains the physiological data, it analyzes the heart rate variability (HRV) and respiratory rate using Fourier Transform. The calculation formula of Fourier Transform is as follows:
[0079]
[0080] Where f(x) is the signal in the time domain; F(t) is the representation in the frequency domain. The physiological state example is shown in Figure 3 .
[0081] After the data processing module obtains the sound wave data, it extracts the sound wave features using Mel Frequency Cepstral Coefficients (MFCC). The calculation formula of Mel Frequency Cepstral Coefficients is as follows:
[0082]
[0083] Where S(m) is the power spectrum, M is the number of filters, and n is the index of the coefficient. The sound example is shown in Figure 4 . Please refer to Figure 2 The feature fusion module 4 performs feature fusion as follows:
[0084] Step A1: Align the expression features, behavior features, physiological features, and sound wave features in the time scale.
[0085] Step A2: Concatenate the expression features, behavior features, physiological features, and sound wave features to form the comprehensive feature vector.
[0086] Step A3: Use dimension reduction techniques to reduce the dimension of the comprehensive feature vector, which can reduce redundant information and improve computational efficiency.
[0087] The feature fusion module 4 obtains the comprehensive feature vector by the following calculation formula:
[0088] X = [F face， F behavior , F heart , F resp , F audio ]
[0089] Wherein, X is the comprehensive feature vector; F face is the expression feature; F behavior is the behavior feature; F heart and F resp is the physiological feature; F audio is the acoustic feature.
[0090] The implementation method of the feature fusion module 4 includes:
[0091] 1. Early fusion:
[0092] The features of each modality are spliced in the extraction stage and input into a unified model.
[0093] 2. Late fusion:
[0094] The features of each modality are input into separate models, and the outputs of the modalities are finally fused.
[0095] Vector shaping:
[0096] The extracted features are combined into the final facial expression vector, which can be used to train a machine learning model for emotion recognition.
[0097] Calculation method:
[0098] 1. Feature combination:
[0099] All geometric features and deep learning features are combined into a vector.
[0100] For example, if there are 30 geometric features and 128 CNN features, the final vector is 158-dimensional.
[0101] 2. Feature standardization:
[0102] The vector is standardized so that the value range of each feature is the same.
[0103] For example, each feature is standardized with a mean of 0 and a standard deviation of 1.
[0104] The above model training and prediction module 5 is trained by the machine learning classification module, which uses support vector machines (SVM), random forests, or deep learning models such as multi-layer perceptrons (MLP) for classification. The calculation formula for optimizing the support vector machine is as follows:
[0105]
[0106] Subject: y i (w·x i +b)≥1-ξ i
[0107] Where ω is the weight vector; x i It is the input feature vector; y i It is a tag; ξ i C is the slack variable; C is the regularization parameter.
[0108] The aforementioned model training and prediction module 5 also uses cross-validation and confusion matrix to evaluate module performance and achieve high accuracy. The formula for calculating accuracy is as follows:
[0109]
[0110] Among them, Accuracy is the accuracy rate; TP is true positive; TN is true negative; FP is false positive; and FN is false negative.
[0111] The steps performed by the model training and prediction module 5 are as follows:
[0112] 1. Train the model using the training dataset (which contains labeled facial expression vectors and corresponding emotion labels).
[0113] 2. Use cross-validation to evaluate the model's performance.
[0114] 3. Input the new facial expression vectors into the trained model for emotion prediction.
[0115] Assuming the calculation of the happiness angle:
[0116] The angle of happiness usually refers to the degree to which the corners of the mouth turn up, which is particularly important in facial expression recognition because it can effectively reflect an animal's happy mood.
[0117] Calculation method:
[0118] 1. Select key points:
[0119] The coordinates of the left corner of the mouth are (x1, y1).
[0120] The coordinates of the right corner of the mouth are (x2, y2).
[0121] The coordinates of the nose tip are (x3, y3).
[0122] Assumption: Left corner of the mouth coordinates (10, 30)
[0123] The coordinates of the right corner of the mouth are (30, 30).
[0124] Nose tip coordinates (20,10)
[0125] 2. Calculate the angle:
[0126] Calculate the angles θ1 from the left corner of the mouth to the tip of the nose and θ2 from the right corner of the mouth to the tip of the nose, as follows:
[0127]
[0128] Based on the above formula, we can calculate:
[0129]
[0130] This simplified example demonstrates the calculation process for the happiness angle vector.
[0131] 3. From a happiness perspective:
[0132] The angle of happiness can be represented by the average of θ1 and θ2, as follows:
[0133]
[0134] The detailed execution functions of the above-mentioned model training and prediction module 5 are as follows:
[0135] 1. Model training and updating:
[0136] 1-1 Model Training: Train the emotion recognition model using the labeled dataset and update the data.
[0137] Model Evaluation 1-2: The training and prediction module 5 updates the data in real time to ensure its accuracy and adaptability.
[0138] 2. Model Deployment and Application:
[0139] 2-1 Model Deployment: Deploy the updated model to the real-time emotion recognition device for real-time application.
[0140] In a preferred embodiment of the present invention, the model training and prediction module 5 is further connected to an intelligent feedback module 52, and the detailed execution function of the intelligent feedback module 52 is as follows:
[0141] 1. Receiving Emotion Recognition Results:
[0142] 1-1 Receives the sentiment prediction results from the model training and prediction module 5 as the basis for feedback.
[0143] 2. Emotional state analysis:
[0144] 2-1 Emotional Assessment: Assess the pet's current emotional state based on the prediction results, such as: happy, anxious, angry, etc.
[0145] 2-2 Anomaly Detection: Detect whether fluctuations in emotional state are abnormal and determine whether a warning is needed.
[0146] 3. Feedback generation:
[0147] 3-1 Warning Message Generation: Generate corresponding warning messages based on the results of anomaly detection, such as vibration or sound alerts, and adjust the warning level.
[0148] 3-2 Suggestion Generation: Based on emotional state and historical data, generate specific behavioral adjustment suggestions and health management suggestions.
[0149] 4. Feedback transmission:
[0150] 4-1 Diverse prompts: Warning information is conveyed on the electronic device 100 through visual displays, sound prompts, and vibration prompts via liquid crystal displays (LCDs) or light-emitting diodes (LEDs).
[0151] 4-2 User Interface: Display detailed feedback and suggestions in module 6 (which can be a mobile application or a web page), making it easy for owners to view and manage their pets' mood and health status.
[0152] The main feature of this invention is that it provides a biological emotion recognition device, which includes a data acquisition module 2, a data processing module 3, a feature fusion module 4, a model training and prediction module 5, and a display module 6. Through various sensing devices, the data is acquired and processed and calculated to enable the owner to obtain accurate emotional responses from their pet, thereby better understanding the pet and adjusting the way they treat it, so as to achieve a more harmonious relationship between the owner and the pet.
[0153] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, any simple modifications and equivalent structural changes made based on the description and drawings of the present invention should also be included within the scope of the present invention and are hereby stated.
[0154] In summary, the above-mentioned biological emotion recognition device of the present invention can effectively achieve its function and purpose when used, thus the present invention is indeed a highly practical invention.
Claims
1. A biological emotion recognition apparatus, characterized by, Comprising: a data acquisition module, which captures facial images and videos of a living being through a camera to obtain expression data, and the camera also simultaneously captures the behavior actions of the living being to obtain behavior data; acquires the heart rate and breathing rate of the living being through a contact sensor or a non-contact sensor to obtain physiological data; records the sound of the living being through a microphone to obtain sound wave data; a data processing module, which obtains the expression data, the behavior data, the physiological data and the sound wave data from the data acquisition module and processes them to extract expression features, behavior features, physiological features and sound wave features; The expression feature extraction further includes extracting feature vectors from labeled facial key points, and these feature vectors are based on geometric features or deep learning-based feature representations: Geometric feature calculation method: Euclidean distance: Calculate the Euclidean distance between two key points; Distance formula: Distance from left ear tip (x1, y1) to right ear tip (x2, y2) Angle: Calculate the angle formed by three key points; Angle formula: Calculation method: Select key points: Left corner of the mouth coordinates (x1, y1); Right corner of the mouth coordinates (x2, y2); Nose tip coordinates (x3, y3); Calculate the angle: Calculate the angle θ1 from the left corner of the mouth to the nose tip and the angle θ2 from the right corner of the mouth to the nose tip, as follows: The deep learning features, as follows: Use a convolutional neural network; Input the living being's facial image into the pre-trained convolutional neural network model; Extract the intermediate layer features of the convolutional neural network to generate a high-dimensional vector; a feature fusion module, which obtains the expression features, the behavior features, the physiological features and the sound wave features from the data processing module and processes them to form a comprehensive feature vector; a model training and prediction module, which obtains the comprehensive feature vector from the feature fusion module, trains the model through a machine learning classification module, and performs emotion prediction on real-time data of the trained model to obtain an emotion prediction data; and a display module, which obtains the emotion prediction data from the model training and prediction module and displays it, or converts the emotion prediction data into emotion prediction text, emotion prediction graphics or emotion prediction animation through a conversion module and displays it.
2. The biological emotion recognition apparatus of claim 1, wherein, The contact sensor used by the data acquisition module to obtain physiological data is a wearable sensor; and the non-contact sensor is an optical sensor.
3. The biological emotion recognition apparatus of claim 1, wherein, After the data processing module obtains the expression data, it performs image processing and expression data feature extraction through a convolutional neural network, and the feature mapping F of the expression data feature extraction is calculated as follows: F = ReLU (W·I + b); Where W is the convolution kernel; I is the input image; b is the bias; ReLU is the activation function.
4. The biological emotion recognition apparatus of claim 1, wherein, After the data processing module obtains the behavior data, it uses a hidden Markov model or a long short-term memory network time series data analysis method to perform behavior feature extraction, and the probability calculation formula of the hidden Markov model is as follows: Where O is the observed sequence; Q is the sequence of states, and λ is the model parameter.
5. The biological emotion recognition apparatus of claim 1, wherein, After the data processing module obtains the physiological data, it analyzes the heart rate variability and breathing rate using Fourier transform, and the calculation formula of the Fourier transform is as follows: Wherein, f(x) is the signal in time domain; F(t) is the representation in frequency domain.
6. The biological emotion recognition apparatus of claim 1, wherein, After the data processing module obtains the acoustic data, the acoustic feature is extracted using the mel frequency cepstral coefficient, and the calculation formula of the mel frequency cepstral coefficient is as follows: Wherein, S(m) is the power spectrum, M is the number of filters, and n is the index of the coefficient.
7. The biological emotion recognition apparatus of claim 1, wherein, The feature fusion module performs the feature fusion step as follows: A1: Time scale alignment of the expression feature, the behavior feature, the physiological feature and the acoustic feature; A2: The expression feature, the behavior feature, the physiological feature and the acoustic feature are spliced into the comprehensive feature vector; and A3: Dimensionality reduction technology is used to reduce the comprehensive feature vector, reduce redundant information and improve computational efficiency.
8. The biological emotion recognition apparatus of claim 1, wherein, The feature fusion module obtains the comprehensive feature vector of the expression feature, the behavior feature, the physiological feature and the acoustic feature through the following calculation formula: X = [F face ,F behavior ,F heart ,F resp ,F audio ] Wherein, X is a comprehensive feature vector; F face is an expression feature; F behavior is a behavior feature; F heart and F resp is a physiological feature; F audio is a sound wave feature.
9. The biological emotion recognition apparatus of claim 1, wherein, The model training and prediction module is trained by the machine learning classification module, which uses support vector machine, random forest or deep learning model for classification, and the calculation formula of the support vector machine is as follows: The subject is: y i (w x i + b) ≥ 1 - ξ i where ω is a weight vector; x i is an input feature vector; y i is a label; ξ i is a slack variable; and C is a regularization parameter.
10. The biological emotion recognition apparatus of claim 1, wherein, The model training and prediction module also evaluates the module performance through cross validation and confusion matrix to form high accuracy, and the calculation formula of the accuracy is as follows: Wherein, Accuracy is the accuracy; TP is true positive; TN is true negative; FP is false positive; FN is false negative.