Complete local emotion estimation device and emotion estimation method

Abstract

To solve the problem that an existing emotion estimation device is limited to cloud dependence and single modal analysis and has problems in real-time property, privacy, and explainability.SOLUTION: To provide a complete local type device which estimates emotions from the expressions and voices of persons and animals in a multimodal manner and outputs Japanese texts and voices in real time by a generation AI. By using the edge AI accelerator and the small-sized arithmetic unit, an Internet-independent and portable configuration is realized, which is suitable for use in fields such as welfare, disasters, and stockbreeding.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an emotion estimation device and method using edge AI that operates in a completely local environment. Specifically, the present invention relates to a device and method that enables emotion understanding and communication support in fields such as welfare, medical care, nursing care, livestock farming, infrastructure management, and disaster evacuation by estimating emotions in real time from the facial expressions and voices of humans and animals (mammals, livestock, disaster relief animals, etc.) and providing feedback of the results in natural language and voice. [Background technology]

[0002] In recent years, AI-based emotion estimation technology has been applied to support coexistence with animals and psychological support for humans. Conventional technologies often use single-modal methods that analyze only the facial expressions or sounds of specific animals, and because they rely on cloud processing, they have issues with privacy, real-time performance, and radio wave environments. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Document 1: JP 2018-174936A Title of Invention: Pet Emotion Determination Device and Method Abstract: This technology discloses technology for estimating emotions through facial expression recognition based on facial images of dogs and cats. Relevance: This technology shares with the present invention the aspect of facial expression-based emotion estimation, but does not disclose integration processing with audio information or processing using a local edge device. Patent Document 2: JP 2019-202330A Title of Invention: Animal Emotion Recognition Method and Device Abstract: This technology relates to a device that uses an AI system to identify emotions based on animal behavior and cries. Relevance: This technology shares with the present invention the aspect of analyzing cries, but does not disclose image processing, multimodal integration, or integration with generative AI. Patent Document 3: JP 2021-129593A Title of Invention: Animal Voice Recognition and Emotion Feedback System Abstract: This technology proposes a method for analyzing animal cries using a smartphone, estimating emotions on the cloud, and providing feedback on the results. Relevance: Although there are similarities in terms of voice emotion recognition, it does not have the real-time and local execution capabilities of the present invention, nor the feedback function of natural language generation using generative AI. [Non-patent literature]

[0004] [Non-Patent Document 1] Paper 1: Title: "Dog emotion recognition using deep learning on facial expression and vocal signals." Presentation: ICPR 2020. Summary: This paper reports on an attempt to classify dog ​​emotions using CNNs and audio spectrograms in combination. Relevance: While the multimodal analysis method is similar to ours, it does not include real-time processing, local execution on an edge device, or natural language generation or voice output using generative AI. Paper 2: Title: arXiv:2201.06573 "Real-time dog mood detection using embedded AI." Summary: This paper presents a technique for detecting dog emotions (moods) using embedded AI. Relevance: While similar to our real-time use of embedded AI, it differs from our integrated and bidirectional notification system, which integrates audio analysis, natural language generation, and voice feedback. Other: While numerous research projects related to pet AI have been published in IEEE and arXiv, a system like ours that integrates multimodal information in real time at the edge and provides easy-to-understand feedback to humans using generative AI has not yet been disclosed. Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional technology has problems such as (1) limited analysis targets and lack of versatility, (2) reliance on the cloud, which raises issues regarding privacy and signal stability, and (3) the explanation of emotions is simple and lacks familiarity. This invention solves the above problems by improving accuracy through integrated estimation of multiple modalities (facial expressions + speech) and by enabling fully local natural language output and speech output. [Means for solving the problem]

[0006] The present invention comprises the following components: Estimating facial expressions of people or animals captured by a camera using CNN Extract MFCC from audio and estimate emotions using machine learning models These are integrated and fed into a local generative AI to generate natural language. TTS audio output and interactive feedback This processing is performed in real time on a computing device such as a Raspberry Pi 5 using the Hailo-8L AI accelerator. Includes a configuration that allows each module to be switched depending on the characteristics of the subject [Effects of the Invention]

[0007] According to the present invention: (1) High accuracy through multimodal integration of facial expressions and voice (2) Fully local processing ensures privacy and real-time performance (3) Generative AI enables natural and friendly explanations (4) Stable operation possible even outside of radio wave range, at disaster sites, welfare facilities, etc. (5) Platform development for humans and animals is possible (6) Versatile thanks to algorithm design that can flexibly adapt to humans and other animals. [Brief explanation of the drawings]

[0008] [Figure 1] prototype [Figure 2] Simple configuration flow [Figure 3] This is the actual GUI screen. [Figure 4] Simple GUI flow [Figure 5]I entered a prompt with Ollama and Gemma. I entered, "You are a pet advisor who is knowledgeable about dog behavior. Please respond with love for dogs and people." The prompt can be set to a general purpose. [Figure 6] Audio analysis [Figure 7] Hailo Pipeline launch, image analysis [Figure 8] Japanese TTS [Figure 9] GUI screen settings DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment of the present invention has the following configuration: Raspberry Pi 5 USB camera (UVC compliant, 720p) USB microphone (16kHz) Hailo-8L accelerator Housing that can be powered by a mobile battery Speaker and LCD display (3.5 inches) YOLO is applied to the images obtained from the camera to detect the subject, and then facial expressions are classified using TFLite CNN. The audio is classified using a scikit-learn model, and both results are input as prompts to a local generated AI (e.g., Gemma 2B, Mistral, LLaMA, Claude, etc.). Provides feedback via TTS. Highly practical for welfare facilities, disaster evacuation shelters, livestock farms, etc. Use any TTS library or engine. [Example]

[0010] The fully local emotion estimation device according to the present invention operates according to the flow shown in Figure 2. Below, we will show a specific implementation example for estimating the emotions of humans and animals. (1) Image and sound capture of the target To capture images of the target human or animal, a UVC-compliant USB camera (e.g., Logitech C920) is used to capture images at a resolution of 1280 x 720 px and a frame rate of 30 fps. For audio input, a USB-connected microphone (e.g., Sony ECM-PC60) is used to record 3 seconds of audio at 16 kHz mono. (2) Processing of emotion estimation AI model Image data is processed by a YOLOv8-based object detection model running on the Hailo-8L accelerator to detect objects (humans or animals) in real time. Only once the detection area is confirmed, facial expression analysis and voice analysis are performed in parallel. (i) Facial Expression Estimation: The target face region is extracted and input to a TensorFlow Lite format CNN model (e.g., mobilenet_v2.tflite). The output labels are set to 6 to 8 types, such as "joy," "anger," "sadness," "anxiety," "relaxation," and "excitement." (ii) Speech emotion estimation: Mel-frequency cepstral coefficients (MFCCs) are extracted from the recorded audio using Python's librosa library, and classification is performed using a random forest classifier (or XGBoost) pre-trained with scikit-learn. (3) Integrated Processing and Natural Language Generation The results of facial expression estimation and vocal emotion estimation are formatted into prompts and input to a locally running generation AI (e.g., Gemma 2B, Mistral 7B, LLaMA, etc.). An example prompt is as follows: The subject's facial expression is "joy" and their tone of voice is "anxiety." Please explain in easy-to-understand Japanese what emotions they are feeling in the situation. The generated natural language sentence (e.g., "I'm feeling a little nervous, but happy") is received and sent to the output. (4) Output processing (TTS and GUI) The generated text is converted into speech using a TTS engine such as OpenJTalk and output in real time through a speaker. The GUI screen also displays facial expressions, speech emotion scores, and the generated sentence. [Industrial Applicability]

[0011] The present invention can be used in a wide range of fields, including welfare, medical care, nursing care, agriculture, livestock farming, disaster evacuation, personal psychological support, education, infrastructure facilities, and animal coexistence support. The sense of security provided by fully local processing and the advanced interactivity provided by generative AI also contribute to improving social significance and well-being. The device and method of the present invention are not limited to dogs, but can also be applied to humans and other animal species. While it is recognized that there are differences in facial muscle structure, vocal frequency characteristics, and emotional expression patterns depending on the target species, the following method ensures versatility. The convolutional neural network (CNN) used for facial expression estimation can be implemented as a switchable model, using image datasets according to the type of subject during training. Similarly, the MFCC+ machine learning classifier used for voice emotion estimation is configured to dynamically switch frequency ranges, feature extraction parameters, and classifier settings according to the subject. When designing prompts for the integrated estimation and generation AI, prompt templates can be adjusted to adapt the emotional output sentences to the target species. In this way, by providing individual model configurations / learning data switching / parameter control mechanisms, it is technically possible to apply a common system configuration to different animal species and humans. [Explanation of symbols]

[0012] Prototype configuration diagram 1. USB camera 2. USB microphone 3. AI assembler 4. Raspberry Pi computing device 5. Bluetooth speaker

Claims

1. A portable device for estimating the emotions of a person or an animal, comprising: (A) A camera as an imaging means; (B) a microphone as a means of collecting sound; (C) an object detection means for detecting an object based on the imaging data; (D) a first emotion estimation means for estimating facial expressions from images; (E) A second emotion estimation means for extracting MFCCs from speech and estimating emotions; (F) an integration means for integrating the results of said first and second estimation means; (G) an output means for outputting the integrated results in natural language and voice using a generative AI model; An emotion estimation device comprising:

2. 2. The emotion estimation device according to claim 1, wherein the generative AI model is implemented on a local device and generates natural language without requiring an internet connection.

3. The emotion estimation device according to claim 1 or 2, wherein the emotion estimation device is used for stress management and communication support by visualizing and audibly expressing the emotions of people or animals in environments such as livestock farming, welfare facilities, disaster evacuation shelters, and educational settings.

Images