Fully local ai watch system and method

Abstract

To provide a system for detecting an aged person, a person in need of nursing care, and a fall accident at a construction site without depending on a cloud, and issuing an alarm in an emergency while giving a psychological sense of security.SOLUTION: The present invention includes a pose estimation means for estimating a skeleton of a person from a camera image in real time and detecting a falling state, a speech synthesis means for speaking in Japanese when a fall is detected, an interaction generation means for recognizing a person's utterance and generating a polite answer using a generative AI model, and an alarm means for sounding a siren when a negative utterance is detected. They operate in a fully local configuration with RaspberryPi5 and Hailo-8L to achieve both privacy protection and low latency. To integrally perform fall detection, psychological care, and alarm issuance, and to exhibit high usefulness in welfare, nursing, and disaster sites.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to safety monitoring and watching technology, and in particular to a cloud-based, completely local AI watching system and method that detects a person's fall in real time, provides reassurance through thoughtful conversation, and issues an alarm. [Background technology]

[0002] Falls among the elderly and those requiring care have become a serious problem both at home and in facilities, and rapid detection and response are essential. Conventional monitoring systems have mainly consisted of wearable devices equipped with acceleration sensors or video surveillance systems connected to cloud servers. However, these systems suffer from delays in cloud communications and a lack of psychological support. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP2019212345A: A device that detects falls using a camera and AI (mainly cloud analysis). JP2020134567A: A voice dialogue robot for welfare (no fall detection). JP2022034560A: Monitoring with IoT sensors (mainly cloud dependent). [Non-patent literature]

[0004] [Non-Patent Document 1] Non-patent document 1: Elderly monitoring system using dynamic image recognition, Grant-in-Aid for Scientific Research Report (Project No. 20K23333), 2023. Non-patent document 2: Real-time risk assessment for preventing patient falls using an AI-based system, Journal of the Japanese Society for Fall Prevention, Vol. 8, No. 3, pp. 5-12, 2021. Non-patent document 3: Fall detection algorithm for elderly people using a triaxial accelerometer, Architectural Institute of Japan Technical Report Collection, Vol. 83, No. 753, pp. 913-920, 2022. Summary of the Invention [Problem to be solved by the invention]

[0005] The objective of this invention is to provide a completely local AI monitoring system that detects a person's fall in real time, provides a sense of security through compassionate conversation, and automatically issues an alarm if necessary. [Means for solving the problem]

[0006] The present invention camera means for capturing an image of a person; a pose estimation means for estimating skeletal keypoints from the acquired image and detecting a falling state; A voice synthesis means that speaks in Japanese, such as "Are you OK?", when a fall is detected; a voice input means for acquiring a person's voice and converting it into text using a voice recognition means; A dialogue generator that uses generative AI models (such as Gemma 2B) to generate reassuring and considerate responses; An alarm that sounds a siren when negative speech (e.g., "help me," "it hurts," "I can't move") is detected. A control method that runs these in real time on local devices such as the Raspberry Pi 5 and Hailo-8L computing devices, without relying on the cloud. Equipped with. [Effects of the Invention]

[0007] Completely local real-time processing prevents communication delays and privacy leaks. It combines fall detection with psychological care, providing a dialogue that gives a sense of security rather than simply alerting the user. An alarm is automatically sounded when negative speech is made, enabling prompt notification of abnormalities. It runs on low-power devices such as the Raspberry Pi 5 computing device and Hailo-8L, making it easy to install in facilities and homes. It can be applied in a wide range of fields, including welfare, nursing care, disaster sites, and construction sites. [Brief explanation of the drawings]

[0008] [Figure 1] Prototype configuration diagram [Figure 2] flow [Figure 3] Enter compassion into the AI ​​prompt DETAILED DESCRIPTION OF THE INVENTION

[0009] Figure 1 As shown in Figure 1, the fully local AI monitoring system of the present invention comprises the following main components: Camera section It is installed indoors or inside a facility and captures the movements and postures of the person being monitored. USB cameras, CSI cameras, etc. can be used. Pose Estimation Unit Extract a person's skeletal key points (shoulders, hips, knees, etc.) from camera images using a deep learning model such as YOLOv8 Pose. Real-time processing is achieved using edge AI accelerators (e.g., Hailo-8L, etc.). Fall detection unit Based on key point information, the relationship between shoulder and waist height, the angle of inclination of the body, and the speed of change in posture are analyzed to determine the state of a fall. Speech synthesis section Using a Japanese speech synthesis engine such as OpenJTalk, when a fall is detected, the system will ask questions such as "Are you OK?" Microphone voice recognition section The monitored person's response is picked up from a Bluetooth microphone or the built-in microphone and converted into text locally using Whisper.cpp or similar. Dialogue Generation Unit Give prompts that convey compassion and affection to local LLMs (e.g., Gemma 2B / Ollama), and generate dialogue responses that provide a sense of security. Speaker alarm output section If the speech recognition result contains negative utterances such as "help me," "it hurts," or "I can't move," a siren sound (siren.wav) is output from the speaker. Control unit The above components are controlled in an integrated manner. A Python script runs on the Raspberry Pi 5 computing device, and all processing is completed entirely locally. Figure 2 As shown in the flowchart, the system operates in the following steps: The camera unit captures an image of the person. The pose estimation unit extracts skeletal information. The fall detection unit detects falls through posture analysis. When you fall, the voice synthesis unit asks, "Are you okay?" The voice recognition unit converts the monitored person's responses into text. If the response is negative, the alarm output unit will sound a siren. If the response is positive, the dialogue generation unit creates a thoughtful response, which the speech synthesis unit plays back. This loop continues in real time. [Example]

[0010] Pose estimation: The YOLOv8 Pose model is accelerated by Hailo-8L to detect 17 human skeleton points in real time. Fall detection: The state of a fall is determined based on the relative positions of the shoulders and hips. Voice dialogue: Using OpenJTalk, a Japanese voice asks, "Are you OK?" Speech recognition: Convert speech into text using Whisper.cpp. AI Response: Set "Caring and Loving" prompts for Gemma 2B (Ollama) to generate kind responses. [Industrial Applicability]

[0011] Our fully local AI monitoring system integrates fall detection, considerate conversation, negative speech detection, and alarm notification, and because it does not rely on the cloud, it can achieve both privacy protection and low latency, making it widely applicable in the following industrial fields: Welfare and nursing care field: Can be introduced as a monitoring system for elderly people living alone and those requiring nursing care. Contributes to preventing accidents within the facility and responding quickly in emergencies. Medical field Used for fall monitoring and psychological care for patients in hospitals and rehabilitation facilities. Can be integrated with existing nurse call and hospital alarm systems. Construction infrastructure field Used to monitor falls and accidents involving workers at construction sites and plants. Applied to emergency detection and alarm issuance at disaster sites and when working at height. Home Safety Management Used as a smart home or personal safety and monitoring device. Combine with IoT devices and home assistants to enhance safety. Disaster prevention and response. In the event of a disaster such as an earthquake or fire, the system automatically detects if a victim falls or collapses, and immediately issues an alarm and calls out to the victim on the scene, helping to mitigate damage. As described above, the present invention is expected to be put to practical use in a wide range of industrial fields, including welfare, medical care, construction, home use, and disaster response, and is therefore extremely useful industrially. [Explanation of symbols]

[0012] 1. USB camera 2. AI Accelerator 3. USB microphone 4. Bluetooth Speaker 5. Raspberry Pi 5 computing device

Claims

1. This is a completely local AI monitoring system that detects a person who has fallen, engages in a considerate dialogue with that person, and issues an alarm if necessary. camera means for capturing an image of a person by a camera; a pose estimation means for estimating skeletal information of the person and determining a state of falling; a dialogue initiation means for speaking to the person in Japanese using a voice synthesis means when the pose estimation means detects a falling state; a voice input means for acquiring the voice of the person and converting it into text by a voice recognition means; A dialogue generation means for generating a thoughtful response using a generative AI model based on the text acquired by the speech recognition means; an alarm means for outputting a siren sound when the text is determined to be a speech requesting help or a negative speech; A control means for executing each of the above means on a local device without relying on a cloud server; A completely local AI monitoring system characterized by the following:

2. In the completely local AI monitoring system according to claim 1, The pose estimation means is a completely local AI monitoring system that runs a YOLO-based pose estimation model at high speed using an edge AI accelerator such as Hailo-8L.

3. In the completely local AI monitoring system according to claim 1 or 2, The dialogue generation means has a prompt setting that generates a response sentence that conveys a sense of security and affection depending on the content of the person's speech, and the voice synthesis means converts the generated response into Japanese speech and plays it back, making it a completely local AI monitoring system.

Images