System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting image data and voice data; means for analyzing the image data and the voice data to detect an abnormality; means for transmitting a notification when an abnormality is detected; means for receiving the notification and notifying a user; and means for making an emergency call when there is no response to the notification.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The number of elderly households living alone has reached 7.42 million nationwide, and there are growing concerns about sudden death, lonely death, and even burglary. Furthermore, it is difficult to manage the health and lifestyle habits of the elderly, so a rapid response is required when an abnormality occurs. In this situation, a system that can detect abnormalities and notify appropriate notifications is needed to provide an environment where the elderly can live with peace of mind. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including a means for collecting image data and voice data, a means for analyzing the image data and voice data to detect abnormalities, a means for sending a notification when an abnormality is detected, a means for receiving the notification and notifying the user, and a means for making an emergency call when there is no response to the notification. Furthermore, by providing a system including a means for periodically providing the user with advice on maintaining health and improving lifestyle habits, an analysis means using image diagnostic technology and voice recognition technology, and an abnormality detection means for identifying voice data such as the sound of a fall or a voice sounding like one in distress, the safety and security of elderly people living alone can be ensured.

[0006] "Image data" refers to visual information captured using a camera or other imaging device.

[0007] "Audio Data" refers to sound information captured using a microphone or other recording device.

[0008] "Means for collecting" refers to a device or method for acquiring image data or audio data and inputting it into the system.

[0009] "Means for analyzing" refers to a device or method for analyzing collected data and detecting anomalies.

[0010] "Detecting anomalies" refers to discovering unusual conditions or events from analyzed data.

[0011] "Means for sending notification" refers to a device or method for communicating information about an abnormality to other devices or people when an abnormality is detected.

[0012] "Means for receiving and notifying a user" refers to a device or method for receiving the transmitted notification and conveying it to the user.

[0013] "Means for making an emergency call" refers to a device or method for contacting emergency services or designated contacts if an abnormality occurs and the user does not respond.

[0014] "Means for periodically providing a user with advice on maintaining health and improving lifestyle habits" refers to a device or method for periodically providing information on health and lifestyle habits to assist the user in their daily lives.

[0015] "Imaging diagnostic technology" refers to technology for analyzing image data and extracting or recognizing specific information.

[0016] "Speech recognition technology" refers to technology for analyzing voice data and understanding or recognizing sounds and words.

[0017] "Falling sounds" refers to the specific sounds that are made when a person falls.

[0018] "Distressed voice" refers to the distinctive vocal characteristics that a person makes when experiencing pain or difficulty. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system that collects image data and audio data, analyzes them to detect abnormalities, and when an abnormality is detected, issues a notification and, if necessary, makes an emergency call.

[0041] System Configuration

[0042] server

[0043] The server analyzes the collected data and determines whether there are any abnormalities. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image diagnostic technology and voice recognition technology.

[0044] Terminal

[0045] The device is installed in the user's home and collects images and sounds on a daily basis. It also receives notifications from the server and conveys them to the user. The device has a built-in camera, microphone, and speaker.

[0046] User

[0047] The user is an elderly person living alone. He interacts with the device during his normal daily life and follows instructions from the device when necessary.

[0048] Program processing

[0049] Server Processing

[0050] 1. Data Reception and Analysis

[0051] The server receives the image data and audio data sent from the terminal. For example, it receives the user's image at a fixed time every day and analyzes the audio data.

[0052] The server uses image diagnostic technology to analyze the user's posture, complexion, etc., and detects abnormalities. For example, if the user falls or looks pale, it will determine that there is an abnormality.

[0053] Using voice recognition technology, the collected voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[0054] 2. Identifying and notifying abnormalities

[0055] The server determines whether there is an abnormality based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device, such as a message saying "A fall has been detected."

[0056] 3. Data recording and analysis

[0057] All data and events are logged and analyzed over the long term, allowing users to understand their health and lifestyle patterns.

[0058] Terminal handling

[0059] 1. Monitoring and Data Transmission

[0060] The device periodically takes pictures and records audio. For example, it takes a picture of the user every morning at 7:00 and sends the data to a server.

[0061] The collected data is sent to a server in real time via Wi-Fi.

[0062] 2. Receiving notifications and notifying users

[0063] When an abnormality notification is received from the server, the device will notify the user by voice, for example, saying, "Are you OK? A fall has been detected."

[0064] If the user does not respond, the device will automatically make an emergency call, for example by automatically dialing 119 to contact emergency services.

[0065] 3. Proposals for maintaining health and improving lifestyle habits

[0066] The device periodically provides users with advice on maintaining their health and lifestyle habits, such as voice messages like "Eat some vegetables today" or "Get outside and do some light exercise."

[0067] Specific examples

[0068] Scenario: User falls

[0069] 1. The user falls in the middle of the night.

[0070] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[0071] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0072] 4. The server immediately sends a notification of the fall accident to the device.

[0073] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[0074] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[0075] In this way, this system can ensure safety and security for elderly people living alone.

[0076] The processing flow will be explained below.

[0077] Server Processing

[0078] Step 1: Receiving data

[0079] The server receives image data and audio data sent from the device. For example, it receives a photo of the user's face, an image of the room, and recorded audio every hour.

[0080] Step 2: Image analysis

[0081] The server analyzes the received image data using image diagnostic technology, for example, analyzing the user's posture and facial expression to determine whether they are normal.

[0082] Step 3: Audio analysis

[0083] Using voice recognition technology, the received voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding like someone is in pain). For example, it checks whether the data contains the sound of a fall.

[0084] Step 4: Identifying anomalies

[0085] The server determines whether or not there is an abnormality based on the results of image and audio analysis. If an abnormality is detected, it proceeds to the next processing step.

[0086] Step 5: Generate notifications

[0087] If an abnormality is detected, the server generates and sends a notification to the device, such as a message saying "A fall has been detected."

[0088] Step 6: Data recording

[0089] The server records all events and analysis results. For example, it saves a log such as "January 10th, fall detected, notification sent."

[0090] Terminal handling

[0091] Step 1: Start monitoring

[0092] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[0093] Step 2: Send data

[0094] The device transmits the collected image and audio data to the server in real time.

[0095] Step 3: Receive notifications

[0096] Receive notifications from the server, for example, a message saying "A fall has been detected."

[0097] Step 4: Notify users

[0098] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[0099] Step 5: Check the response

[0100] Check the user's response, for example, "It's okay."

[0101] Step 6: Call Emergency Services

[0102] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[0103] Step 7: Regular advice

[0104] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[0105] User Behavior

[0106] Step 1: Living a normal life

[0107] The user goes about their daily life as normal.

[0108] Step 2: Voice interaction

[0109] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[0110] Step 3: Responding to emergencies

[0111] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[0112] Step 4: Stay healthy and take advice

[0113] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[0114] The above is the specific processing flow of the system. This system can provide safety and security to elderly people living alone.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] Elderly people living alone need to be able to quickly detect abnormalities in their daily lives, such as falls or sudden illness, and respond appropriately. However, conventional monitoring systems have a high risk of delaying emergency response due to low accuracy in detecting abnormalities and delayed notification. Furthermore, preventive health management is difficult due to a lack of support for understanding long-term health conditions and improving lifestyle habits. The purpose of this invention is to provide a system that solves these problems and improves the safety and security of elderly people living alone.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes a means for receiving image data and audio data, a means for analyzing the image data and audio data to detect an abnormality, and a means for sending a notification when an abnormality is detected, thereby enabling the abnormality to be detected quickly and accurately and notified to the user in real time.

[0120] "Image data" is digital data that includes visual information captured by a camera or the like.

[0121] "Audio data" is digital data that includes sound information collected by a microphone or the like.

[0122] "Means for detecting anomalies" refers to technologies and methods for analyzing image and audio data to detect events that deviate from the user's normal behavioral patterns or states (e.g., falls, distressed voices, etc.).

[0123] "Means for sending notifications" refers to techniques or methods for communicating information about an abnormality to other devices or users when that information is detected.

[0124] "Means for notifying the user" refers to techniques or methods for notifying the user of the received notification, including voice messages and display screens.

[0125] "Means for making an emergency call" refers to the technology or method for contacting emergency services (e.g., ambulance, police) if the user does not respond to the abnormality notification.

[0126] "Means for recording data and conducting long-term analysis" refers to techniques and methods for storing collected data and analyzing a user's health status and lifestyle patterns over time.

[0127] "Image processing technology" refers to technology for analyzing digital images, and includes methods such as facial recognition, pose estimation, and object detection.

[0128] "Speech analysis technology" refers to technology for analyzing voice data, and includes methods such as voice recognition, acoustic feature extraction, and abnormal sound detection.

[0129] This invention is a system that collects image and voice data, analyzes them to detect abnormalities, notifies users when an abnormality is detected, and sends an emergency call if necessary. This system ensures the safety and security of elderly people living alone.

[0130] System Configuration

[0131] server

[0132] The server analyzes the collected data and determines whether there are any abnormalities. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image processing and voice analysis technologies. Specifically, the server uses an image processing library such as OpenCV to analyze the user's posture and facial expression, and uses voice recognition technology such as Google Cloud Speech-to-Text to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[0133] Terminal

[0134] The device is installed in the user's home and collects images and audio on a daily basis. The device has a built-in camera, microphone, and speaker, which are used to collect data and send it to a server. The device communicates with the server via Wi-Fi and transmits the collected data in real time. It also receives notifications from the server and relays them to the user. If an abnormality is detected, the device notifies the user by voice, and if the user does not respond, it makes an emergency call.

[0135] User

[0136] The user is an elderly person living alone. They interact with the device as they go about their daily lives and follow instructions from the device when necessary. For example, the device may provide advice such as "Go outside and do some light exercise."

[0137] Specific examples of data reception and analysis

[0138] For example, if a user falls in the middle of the night, the device will detect an abnormal sound (the sound of a fall). When that data is sent to the server, the server analyzes the received audio data and determines that it is likely a fall accident. The server immediately sends a notification of the fall accident to the device. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response. If the user does not respond, the device will automatically dial "119" to contact emergency services.

[0139] Example prompts for generative AI models

[0140] Please explain how the server and device work together to respond when the user falls, including detailed processing steps and specific actions.

[0141] In this way, the present invention provides a system that uses a high-performance server and terminals to ensure safety and security for elderly people living alone.

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] Step 1:

[0144] The device periodically collects image and audio data using a camera and microphone. The input is a user's photo and surrounding audio, and the output is a digital image file and audio file. For example, a device may take a photo of the user and record their audio every morning at 7:00. Specifically, the device's camera takes a picture and the microphone starts recording.

[0145] Step 2:

[0146] The device sends the collected image and audio data to the server via Wi-Fi. The input is the image and audio files stored on the device, and the output is the data packets sent to the server. Specifically, the device checks for connection to the Wi-Fi network and transfers the files to the server using a data transmission protocol (e.g., FTP).

[0147] Step 3:

[0148] The server performs a consistency check on the image and audio data received from the terminal and begins analysis. The input is the received image and audio files, and the output is data ready for analysis. Specifically, the server records a data reception log and verifies the file format.

[0149] Step 4:

[0150] The server uses OpenCV to analyze image data and detect the user's posture and facial color. The input is an image file, and the output is a repository of analysis results. The server first performs face recognition and then runs a posture estimation algorithm, for example, by detecting contours to determine whether the user's posture is normal or abnormal.

[0151] Step 5:

[0152] The server uses Google Cloud Speech-to-Text to analyze audio data and detect abnormal sounds. The input is an audio file, and the output is the audio data converted into text and the results of abnormal sound detection. Specifically, the server extracts features from the audio waveform and detects, for example, sounds of falling or sounds of distress.

[0153] Step 6:

[0154] The server integrates the analysis results of the image data and audio data to determine whether or not an abnormality has occurred. The inputs are the image analysis results and audio analysis results, and the output is the abnormality detection judgment result. Specifically, the server executes the abnormality detection algorithm and makes a judgment such as "A fall has been detected."

[0155] Step 7:

[0156] The server sends a notification to the terminal when an abnormality is detected. The input is the result of the abnormality detection judgment, and the output is a notification message to the terminal. In concrete terms, the server generates the notification message and sends it to the terminal via the network.

[0157] Step 8:

[0158] The device receives the notification from the server and notifies the user by voice. The received notification message is the input, and the voice message is generated as the output. Specifically, the device calls out to the user through the speaker, saying, "Are you OK? A fall has been detected."

[0159] Step 9:

[0160] If the user does not respond, the device automatically makes an emergency call. The input is the information that the user did not respond, and the output is a call to emergency services. Specifically, the device automatically dials "119" and notifies emergency services of the accident situation.

[0161] (Application example 1)

[0162] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0163] In recent years, there has been a growing emphasis on strengthening home security and improving the safety of home dwellers, including the elderly and those living alone. However, conventional security systems often lack a rapid or efficient response when an abnormality occurs. A particular issue is the lack of real-time abnormality detection and emergency reporting. The present invention aims to solve these issues by providing a system that instantly detects abnormalities in the home, notifies the user, and responds quickly.

[0164] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0165] In this invention, the server includes means for collecting image data and audio data, means for analyzing the image data and audio data to detect abnormalities, means for analyzing data from sensors installed in the home in real time, means for sending real-time notifications to a smartphone based on the analysis results, and means for storing the data for later analysis and recording, thereby enabling immediate detection of abnormal situations in the home and prompt notification or emergency calls.

[0166] "Image data" refers to visual information acquired using an optical sensor and recorded as digital data.

[0167] "Audio data" refers to acoustic information collected through an acoustic sensor such as a microphone and recorded as digital data.

[0168] A "sensor" is a device that detects a physical or chemical phenomenon and outputs it as an electrical signal.

[0169] "Analysis" is the process of calculation and evaluation used to process collected data and find specific patterns or anomalies.

[0170] An "anomaly" is an event or condition that deviates from the normal or expected pattern.

[0171] A "notification" is a message or signal that notifies the user that an abnormality has been detected.

[0172] "Emergency notification" is a means of automatically sending a warning to pre-defined contacts when an abnormality occurs and the user does not respond.

[0173] "Real time" refers to the time range in which processing and notification occur with almost no delay after an event occurs.

[0174] "Storage" refers to saving the data used in the analysis and its results for future reference or analysis.

[0175] A "smartphone" is a portable information terminal that has the form of a mobile phone but has advanced computing power and can use a variety of applications.

[0176] This invention is a security system for ensuring safety within the home, which collects and analyzes image and audio data to detect abnormalities, notifies the user if an abnormality is detected, and makes an emergency call if there is no response from the user. Furthermore, by analyzing data in real time, it is possible to quickly detect and notify abnormalities.

[0177] server

[0178] The server is equipped with a high-performance processor and large-capacity storage, and performs data analysis using image diagnostics and speech recognition technologies. Specifically, it uses Python and its library, OpenCV, to perform image analysis, TensorFlow to run machine learning models, and Google Cloud Speech-to-Text API to transcribe and analyze voice data.

[0179] The server uses the following hardware and software:

[0180] Hardware: High-performance processors, server storage

[0181] Software: Python, OpenCV, TensorFlow, Google Cloud Speech-to-Text API

[0182] Terminal

[0183] The devices installed in homes are equipped with built-in cameras and microphones and periodically collect image and audio data. This collected data is then sent to a server in real time via Wi-Fi, minimizing the delay before the data reaches the server, achieving high real-time performance.

[0184] User

[0185] If an abnormality is detected, the user will be notified on their smartphone. This notification is sent as a push notification using Firebase Cloud Messaging. If the user does not respond to the notification, an emergency SMS will be sent to preset emergency contacts using the Twilio API.

[0186] Specific examples

[0187] For example, if a user hears the sound of an intruder breaking a window into their home late at night, a sensor installed in the home will detect this abnormal sound and send the data to a server in real time. The server will analyze the audio data and determine that an abnormal sound (intrusion sound) has been detected. At this time, the server will immediately send a push notification to the user's smartphone saying, "An abnormal sound has been detected. Please check the safety of your home." If the user does not respond to the notification, the app will automatically send an SMS to the designated emergency contact.

[0188] Specific examples of software and hardware used

[0189] Hardware: Home cameras, microphones, servers with high-performance processors, and users' smartphones

[0190] Software: Python, OpenCV, TensorFlow, Google Cloud Speech-to-Text API, Firebase Cloud Messaging, Twilio API

[0191] Prompt Sentence Examples

[0192] Describe the design of a smart security app that analyzes and notifies users in real time if any abnormal sounds (screams, breaking glass, etc.) are detected in their home.

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] The device is installed in the home and periodically collects image and audio data using a camera and microphone. The camera takes pictures at scheduled intervals, and the microphone records continuously. The collected data is sent to a server in real time via Wi-Fi.

[0196] Input: Real-time image and audio data from cameras and microphones

[0197] Output: Image and audio data transmitted via Wi-Fi

[0198] Step 2:

[0199] The server receives the image data and audio data sent from the device. The received data is read into the Python program to analyze it.

[0200] Input: Image and audio data received via Wi-Fi

[0201] Output: Image and audio data loaded for analysis

[0202] Step 3:

[0203] The server performs image analysis. It processes the image data using the OpenCV library and determines whether there are any abnormalities (e.g., a person lying down, a suspicious person entering, etc.).

[0204] Input: Imported image data

[0205] Data processing: Image processing using OpenCV, such as face recognition and motion detection

[0206] Output: Analysis results regarding the presence or absence of abnormalities

[0207] Step 4:

[0208] The server performs voice analysis. It transcribes the voice data using the Google Cloud Speech-to-Text API and then analyzes it. If any abnormal sounds are detected (e.g., screaming, breaking glass), the details are recorded.

[0209] Input: Loaded audio data

[0210] Data processing: Speech transcription and analysis using Google Cloud Speech-to-Text API

[0211] Output: Analysis results for abnormal sounds

[0212] Step 5:

[0213] If the server detects an anomaly, it generates a real-time notification based on the analysis results and sends a push notification to the smartphone using Firebase Cloud Messaging.

[0214] Input: Analysis results regarding the presence or absence of abnormalities

[0215] Data processing: Notification message generation

[0216] Output: Push notification to the user's smartphone

[0217] Step 6:

[0218] The user receives a notification and checks it on their smartphone. They read the notification message and check whether there are any abnormalities. They follow instructions such as "Make sure you are safe."

[0219] Input: Push notification to smartphone

[0220] Output: User response (confirmation or no response)

[0221] Step 7:

[0222] The server monitors the response to the notification. If there is no response from the user within a certain time, it will execute emergency notification procedures and send an SMS to the emergency contact using the Twilio API.

[0223] Input: User response (no response)

[0224] Data processing: Emergency call message generation

[0225] Output: SMS notification to emergency contacts

[0226] Step 8:

[0227] The server stores the data, recording the received image data, audio data, and analysis results in a database for later analysis.

[0228] Input: Analysis results, original data

[0229] Data processing: Data recording

[0230] Output: Stored database entries

[0231] Through these steps, the present invention quickly detects abnormal situations within the home and provides appropriate notification and emergency response.

[0232] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0233] This invention provides a system that collects image and audio data, analyzes them to detect abnormalities, notifies users when an abnormality is detected, and makes emergency calls if necessary.Furthermore, by incorporating an emotion engine that recognizes the user's emotions and takes appropriate action based on those emotions, it is possible to provide a higher level of safety and security.

[0234] System Configuration

[0235] server

[0236] The server analyzes the collected data and determines whether there are any abnormalities. It also uses an emotion engine to recognize the user's emotional state. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image diagnosis technology, voice recognition technology, and emotion analysis technology.

[0237] Terminal

[0238] The device is installed in the user's home and collects images and sounds on a daily basis. It also receives notifications from the server and conveys them to the user. The device has a built-in camera, microphone, and speaker.

[0239] User

[0240] The user is an elderly person living alone. He interacts with the device during his normal daily life and follows instructions from the device when necessary.

[0241] Program processing

[0242] Server Processing

[0243] 1. Data Reception and Analysis

[0244] The server receives the image data and audio data sent from the terminal. For example, it receives the user's image at a fixed time every day and analyzes the audio data.

[0245] The server uses image diagnostic technology to analyze the user's posture, complexion, etc., and detects abnormalities. For example, if the user falls or looks pale, it will determine that there is an abnormality.

[0246] Using voice recognition technology, the collected voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[0247] 2. Emotion analysis

[0248] The emotion engine analyzes the user's emotional state from image and voice data, determining, for example, whether the user is feeling stressed or happy.

[0249] 3. Identifying and notifying abnormalities

[0250] The server determines whether or not there are any abnormalities based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device, generating a message such as "A fall has been detected" or "The user is in a stressful state."

[0251] 4. Data recording and analysis

[0252] The server records all data and events as logs and performs long-term analysis, which allows the system to understand not only the user's health status and lifestyle patterns, but also emotional fluctuations.

[0253] Terminal handling

[0254] 1. Start monitoring

[0255] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[0256] 2. Data Transmission

[0257] The device transmits the collected image and audio data to the server in real time.

[0258] 3. Receiving notifications

[0259] Receive notifications from the server, for example, a message saying "A fall has been detected."

[0260] 4. Notice to Users

[0261] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[0262] 5. Response Acknowledgment

[0263] Check the user's response, for example, "It's okay."

[0264] 6. Emergency call

[0265] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[0266] 7. Providing regular advice

[0267] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[0268] User Behavior

[0269] 1. Normal life

[0270] The user goes about their daily life as normal.

[0271] 2. Voice Dialogue

[0272] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[0273] 3. Response in the event of an emergency

[0274] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[0275] 4. Stay healthy and take advice

[0276] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[0277] Specific examples

[0278] Scenario 1: User falls

[0279] 1. The user falls in the middle of the night.

[0280] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[0281] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0282] 4. The server immediately sends a notification of the fall accident to the device.

[0283] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[0284] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[0285] Scenario 2: The user is stressed

[0286] 1. The user is feeling stressed during the day.

[0287] 2. The device sends the captured images and recorded audio to the server.

[0288] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[0289] 4. The server sends a notification to the device saying "The user is in a stressed state."

[0290] 5. The device will advise the user by voice, "Take a deep breath to relax."

[0291] In this way, this system not only ensures the safety and security of elderly people living alone, but also monitors their emotional state, allowing for more comprehensive care.

[0292] The processing flow will be explained below.

[0293] Server Processing

[0294] Step 1: Receiving data

[0295] The server receives image data and audio data sent from the device. For example, it receives a photo of the user's face, an image of the room, and recorded audio every hour.

[0296] Step 2: Image analysis

[0297] The server analyzes the received image data using image diagnostic technology, for example, analyzing the user's posture (whether they have fallen) and facial expression (whether they look distressed) to determine whether there are any abnormalities.

[0298] Step 3: Audio analysis

[0299] Using voice recognition technology, the received voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding like someone is in pain). For example, it checks whether the data contains the sound of a fall.

[0300] Step 4: Sentiment Analysis

[0301] The emotion engine analyzes the user's emotional state from image and audio data, determining, for example, whether the user is feeling stressed or happy.

[0302] Step 5: Identifying anomalies

[0303] The server comprehensively evaluates the results of image analysis, audio analysis, and emotion analysis to determine whether or not there is an abnormality. For example, if a fall is detected and the user sounds distressed, it will determine that there is an abnormality.

[0304] Step 6: Generate notifications

[0305] If an abnormality is detected, the server generates and sends a notification to the device, such as a message saying "A fall has been detected" or "The user is in a stressed state."

[0306] Step 7: Data recording

[0307] The server records all data and events as logs for long-term analysis. For example, it stores logs such as "January 10th, 12:00, fall detected, notification sent."

[0308] Terminal handling

[0309] Step 1: Start monitoring

[0310] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[0311] Step 2: Send data

[0312] The device transmits the collected image and audio data to the server in real time.

[0313] Step 3: Receive notifications

[0314] Receive notifications from the server, for example, a message saying "A fall has been detected."

[0315] Step 4: Notify users

[0316] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[0317] Step 5: Check the response

[0318] Check the user's response, for example, "It's okay."

[0319] Step 6: Call Emergency Services

[0320] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[0321] Step 7: Regular advice

[0322] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[0323] User Behavior

[0324] Step 1: Living a normal life

[0325] The user goes about their daily life as normal.

[0326] Step 2: Voice interaction

[0327] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[0328] Step 3: Responding to emergencies

[0329] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[0330] Step 4: Stay healthy and take advice

[0331] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[0332] Specific examples

[0333] Scenario 1: User falls

[0334] Step 1: The user falls during the night.

[0335] Step 2: The device detects an abnormal sound (falling sound) and sends the data to the server.

[0336] Step 3: The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0337] Step 4: The server immediately sends a notification of the fall accident to the device.

[0338] Step 5: The device asks the user, "Are you OK? Did you fall?" and waits for a response.

[0339] Step 6: If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[0340] Scenario 2: The user is stressed

[0341] Step 1: The user experiences stress during the day.

[0342] Step 2: The device sends the captured images and recorded audio to the server.

[0343] Step 3: The server analyzes the received data and the emotion engine determines that the user is feeling stressed.

[0344] Step 4: The server sends a notification to the device saying "The user is in a stressed state."

[0345] Step 5: The device will advise the user by voice, "Take a deep breath to relax."

[0346] In this way, this system not only ensures the safety and security of elderly people living alone, but also monitors their emotional state, allowing for more comprehensive care.

[0347] Example 2

[0348] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0349] Ensuring safety and security is extremely important for elderly people living alone. However, conventional anomaly detection systems only target physical abnormalities and lack the functionality to monitor the user's emotional state and respond to changes. As a result, it has been difficult to alleviate the psychological burden, loneliness, and stress felt by elderly people. Furthermore, emergency responses are often inappropriate, requiring rapid action. To solve these issues, a system incorporating advanced analysis technology and an emotion engine is needed.

[0350] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0351] In this invention, the server includes means for analyzing image data and audio data to detect abnormalities, means for analyzing the user's emotional state from the image data and audio data, and means for sending a notification when an abnormality is detected. This makes it possible to monitor not only physical abnormalities but also the user's emotional state. Specifically, by using emotion analysis technology in addition to image diagnosis technology and voice recognition technology, more accurate abnormality detection and emotion recognition can be achieved. Furthermore, by periodically providing advice on maintaining health and improving lifestyle habits, it is possible to improve the quality of life of elderly people. This provides a system that allows elderly people living alone to continue living more safely and with peace of mind.

[0352] "Image data" is a digital representation of visual information acquired by a photographic device such as a camera.

[0353] "Audio data" refers to a digital representation of sound information captured by a sound collection device such as a microphone.

[0354] "Means for detecting abnormalities" refers to the technology and process of analyzing image and audio data to detect abnormal events that deviate from normal life patterns.

[0355] "Means for analyzing emotional state" refers to the technology and process for recognizing and determining a user's emotional state and changes in emotions based on image and audio data.

[0356] "Means for sending notifications" refers to the techniques and processes that appropriately convey information to a terminal or other device when an abnormality or change in emotional state is detected.

[0357] "Means for receiving notifications and notifying users" refers to the technology and process for receiving notification information sent from a server or center and notifying the user of it by voice or other output means.

[0358] "Means for making emergency calls" refers to the technology and processes that automatically contact emergency services (e.g., an ambulance) if the user does not respond.

[0359] "Means for providing advice on maintaining health and improving lifestyle habits" refers to technologies and processes that periodically provide appropriate improvement measures and advice based on the user's health condition and lifestyle habits.

[0360] "Image diagnostic technology" is a technology that analyzes collected image data and recognizes various patterns and features to make a diagnosis.

[0361] "Speech recognition technology" is a technology that analyzes collected voice data and recognizes the content and characteristics of the voice.

[0362] "Emotion analysis technology" is a technology that analyzes and judges a user's emotional state based on image and audio data.

[0363] This invention relates to a system that collects and analyzes image and audio data to detect abnormalities and, if necessary, sends an emergency call. The purpose of this invention is to provide safety and security, particularly for elderly people living alone.

[0364] System Configuration

[0365] server

[0366] Servers are equipped with powerful processors and large storage capacity. They primarily use the following technologies and software:

[0367] OpenCV and TensorFlow are used as image diagnostic technologies.

[0368] Google Cloud Speech-to-Text and Amazon Transcribe are used as voice recognition technologies.

[0369] IBM Watson Tone Analyzer is used as the sentiment analysis technology.

[0370] This allows the collected image and audio data to be analyzed to determine the user's posture, facial color, abnormal sounds, and emotional state.

[0371] Terminal

[0372] The devices are installed in elderly people's homes and collect data on a daily basis. Specifically, they include the following hardware:

[0373] Camera module (e.g. Raspberry Pi camera module)

[0374] microphone

[0375] speaker

[0376] The device uses Python scripts to operate this hardware, periodically collecting images and audio and sending them to the server.

[0377] User

[0378] The user is an elderly person living alone, and can live a normal daily life without any special operations. The user interacts with the device in response to voice prompts.

[0379] Example

[0380] Scenario 1: User falls

[0381] 1. The user falls in the middle of the night.

[0382] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[0383] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0384] 4. The server immediately sends a notification of the fall accident to the device.

[0385] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[0386] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[0387] Scenario 2: The user is stressed

[0388] 1. The user is feeling stressed during the day.

[0389] 2. The device sends the captured images and recorded audio to the server.

[0390] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[0391] 4. The server sends a notification to the device saying "The user is in a stressed state."

[0392] 5. The device will advise the user by voice, "Take a deep breath to relax."

[0393] Prompt Sentence Examples

[0394] The following prompts can be used to generate analysis using a generative AI model:

[0395] "Analyze images of users falling and detect abnormalities."

[0396] "Analyze audio data containing the sound of a fall and detect any abnormalities."

[0397] "Analyze whether the user is in a stressful state."

[0398] This invention provides a system that allows elderly people living alone to continue living more safely and securely. By using a specific combination of hardware and software, highly accurate anomaly detection and emotion recognition become possible.

[0399] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0400] Program processing steps

[0401] Server Processing Steps

[0402] Step 1: Receiving data

[0403] The server receives image data (e.g., JPEG images) and audio data (e.g., WAV audio files) sent from the device. It receives the image data and audio data as input and stores them in temporary storage as output. After this storage, it is ready for analysis.

[0404] Step 2: Image analysis

[0405] The server analyzes the received image data using image diagnosis technology (OpenCV or TensorFlow). Specifically, it uses a face detection algorithm to identify the position of the face, and then analyzes facial color and posture to detect abnormalities. It receives image data as input and obtains analysis results (detection information on abnormal areas) as output.

[0406] Step 3: Audio analysis

[0407] The server converts the received voice data into text using speech recognition technology (Google Cloud Speech-to-Text or Amazon Transcribe), and then detects abnormal sounds. Specifically, it uses a deep learning model to identify sounds of falling or voices that sound like they're in pain. It receives voice data as input and obtains text data and the results of abnormal sound detection as output.

[0408] Step 4: Sentiment Analysis

[0409] The server uses an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotional state from image and voice data. Specifically, it analyzes voice tone and changes in facial expressions to determine stress and happiness. It receives image and voice data as input and obtains the analysis results of the emotional state as output.

[0410] Step 5: Identifying and notifying abnormalities

[0411] The server determines whether or not there is an abnormality based on the analysis results, and if an abnormality is detected, sends a notification to the device. It receives the analysis results as input, generates a notification message (e.g., "A fall has been detected") as output, and sends it to the device.

[0412] Step 6: Record and analyze data

[0413] The server logs all data and events into a non-relational database (MongoDB or Cassandra) for long-term analysis. It receives analysis data and event information as input and stores log data as output.

[0414] Terminal processing steps

[0415] Step 1: Start monitoring

[0416] The device periodically takes pictures and records audio using the built-in camera and microphone. Specifically, it takes pictures and records audio at 7:00 a.m. and 5:00 p.m. every day. The input is a time trigger, and the output is image data and audio data.

[0417] Step 2: Send data

[0418] The terminal transmits the collected image data and audio data to the server in real time. It receives the image data and audio data as input and saves a log of the transmission completion as output.

[0419] Step 3: Receive notifications

[0420] The device receives notifications from the server in real time via WebSocket or Firebase Cloud Messaging. It receives notification messages from the server as input and prepares the notification content as output.

[0421] Step 4: Notify users

[0422] The device uses a speech synthesis engine (such as Google Text-to-Speech) to communicate the received notification to the user aloud. Specifically, it plays a message such as "Are you OK? A fall has been detected" from the speaker. The device receives the notification message as input and plays the audio message as output.

[0423] Step 5: Check the response

[0424] The device collects the user's response using a microphone and analyzes it using a speech recognition engine. Specifically, it converts responses such as "It's okay" into text and sends it to the server. It receives voice data as input, generates text data as output, and sends it to the server.

[0425] Step 6: Call Emergency Services

[0426] If the user does not respond, the device automatically makes an emergency call. Specifically, it uses a module to connect to a telephone line and automatically dials "119." It receives the response confirmation result as input and sends the content of the call (e.g., "The user did not respond, please call an ambulance") as output.

[0427] Step 7: Regular advice

[0428] The device periodically provides users with voice advice on maintaining their health and improving their lifestyle. Specifically, it plays messages such as "Eat your vegetables today" from the speaker. Time is the trigger input, and the voice message is played as the output.

[0429] User action steps

[0430] Step 1: Living a normal life

[0431] Users can continue their daily lives as usual, and no special operations are required.

[0432] Step 2: Voice interaction

[0433] It interacts with the user in response to a prompt from the device. For example, the device asks, "How are you today?" and the user responds, "I'm fine." It receives the question from the device as input and responds as output.

[0434] Step 3: Responding to emergencies

[0435] When a user receives an abnormality notification from the device, they respond by following the instructions. For example, when asked "Did you fall?", they respond "Yes, I fell." The device receives a question from the device as input and responds as output.

[0436] Step 4: Stay healthy and take advice

[0437] Accept suggestions from the device for maintaining health and improving lifestyle habits. For example, try to maintain your health by following the meal menu suggested by the device. Receive suggestions from the device as input and execute them as output.

[0438] (Application example 2)

[0439] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0440] Elderly people living alone need a system that can not only detect falls and changes in their health, but also monitor their emotional state and take appropriate action. However, while current systems can detect abnormalities and issue emergency notifications, they do not take action based on emotional state, which means that they are unable to respond adequately when elderly people feel stressed or anxious.

[0441] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0442] In this invention, the server includes means for collecting image data and voice data, means for analyzing the image data and voice data to detect abnormalities, means for sending a notification when an abnormality is detected, means for receiving the notification and notifying the user, means for making an emergency call when there is no response to the notification, and means for analyzing the emotional state of the user using an emotion engine and taking appropriate measures as necessary. This ensures the safety and security of elderly people living alone and enables appropriate measures to be taken according to their emotional state.

[0443] "Image data" is digital data of visual information acquired using a photographing device such as a camera.

[0444] "Audio data" refers to digital data of auditory information acquired using a recording device such as a microphone.

[0445] "Analysis" is the process of using algorithms and programs to evaluate acquired image and audio data and detect specific patterns or anomalies.

[0446] An "abnormality" refers to a phenomenon that differs from normal conditions or behavior and requires an emergency response.

[0447] "Notification" refers to a warning or information message sent to the user when an abnormality is detected.

[0448] An "emergency call" is an action that automatically contacts emergency services or relevant parties when there is no response from the user.

[0449] An "emotion engine" is a control device or program that analyzes image data and audio data to recognize the user's emotional state and take appropriate action.

[0450] "Advice on maintaining health and improving lifestyle habits" refers to specific guidance and suggestions to help users live a healthy life.

[0451] "Image diagnostic technology" is a technology that analyzes image data to evaluate a user's posture, facial expression, and health condition.

[0452] "Voice recognition technology" is a technology for analyzing voice data to evaluate the content of a user's speech and the acoustic conditions.

[0453] This invention provides a system that collects image and audio data, analyzes them to detect abnormalities, and sends notifications and emergency calls as necessary.Furthermore, by using an emotion engine to recognize the user's emotional state and take appropriate measures, it is possible to provide users with a high level of safety and security.

[0454] System Configuration

[0455] Hardware:

[0456] 1. Camera: A device installed in the home that periodically captures images. An example is a network-enabled camera.

[0457] 2. Microphone: A device that collects sounds within the home. A highly sensitive microphone is used.

[0458] 3. Server: A device with a high-performance processor and large-capacity storage for analyzing data. It is mainly provided by AWS (Amazon Web Services) or on-premise high-performance computers.

[0459] 4. Terminal: A device installed in the home that communicates with the server. This includes smartphone apps and dedicated devices.

[0460] software:

[0461] 1. Flask: A server-side web application framework that allows the server to receive and process data.

[0462] 2. OpenCV: A library for image processing. It is used to detect anomalies in images.

[0463] 3. TensorFlow: A machine learning library for image and audio analysis, used for anomaly detection and sentiment analysis.

[0464] Server Roles

[0465] The server receives image and audio data sent from the device, analyzes it, and determines whether there are any abnormalities or the user's emotional state. The server has the following functions:

[0466] Data reception and analysis: Image and voice data are received and analyzed in real time. Image diagnostic and voice recognition technologies are used to evaluate the user's posture, complexion, tone of voice, etc.

[0467] Anomaly detection and notification: If an anomaly is detected, a notification is generated and sent to the device immediately, for example, a message saying "Fall detected."

[0468] Emotion analysis: Using an emotion engine to analyze a user's emotional state, for example, determining whether they are feeling stressed or happy.

[0469] Device Role

[0470] The terminal is installed in the home and communicates with the server. It has the following functions:

[0471] Monitoring and data transmission: Take pictures and record audio periodically and transmit them to a server. For example, run these at 7am and 5pm every day.

[0472] Notification reception and user notification: Receives notifications from the server and notifies the user by voice or text. For example, an audio message saying "A fall has been detected" is played.

[0473] Emergency Calls: Automatically make an emergency call if the user does not respond, for example by automatically dialing an emergency number and contacting emergency services.

[0474] Specific examples

[0475] Scenario 1: User falls

[0476] 1. The user falls in the middle of the night.

[0477] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[0478] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0479] 4. The server immediately sends a notification of the fall accident to the device.

[0480] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[0481] 6. If the user does not respond, the device automatically makes an emergency call. The device automatically dials the emergency number and contacts emergency services.

[0482] Scenario 2: The user is stressed

[0483] 1. The user is feeling stressed during the day.

[0484] 2. The device sends the captured images and recorded audio to the server.

[0485] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[0486] 4. The server sends a notification to the device saying "The user is in a stressed state."

[0487] 5. The device will advise the user by voice, "Take a deep breath to relax."

[0488] Example prompt for a generative AI model:

[0489] Take the user's image and audio data as input, analyze it for abnormalities and emotional state, notify the user if an abnormality is detected, and provide advice on relaxation if the user is in a stressful state.

[0490] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0491] Step 1:

[0492] The device periodically takes pictures and records audio. For example, it does this every morning at 7:00 and evening at 5:00. The captured image data and recorded audio data are the input to the device. The device collects this data and prepares it for transmission. The output is image data and audio data.

[0493] Step 2:

[0494] The terminal sends the collected image data and audio data to the server. The transmission is done in real time. The image data and audio data output from the terminal become the input to the server. The data is transferred via the Internet. The server receives this data. The output is the received image data and audio data.

[0495] Step 3:

[0496] The server analyzes the image and audio data it receives. For the image data, OpenCV is used to evaluate the user's posture and facial color. For the audio data, TensorFlow is used to analyze abnormal sounds (such as the sound of a fall or a voice sounding distressed). The input is image data and audio data, and the output is the anomaly detection result.

[0497] Step 4:

[0498] The server uses an emotion engine to analyze the user's emotional state from the received image and audio data. For example, it determines whether the user is stressed or relaxed. The input is image and audio data, and the output is the emotion analysis result. The emotion engine uses a pre-trained generative AI model.

[0499] Step 5:

[0500] The server determines whether or not there is an abnormality based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device. For example, it generates a message saying "A fall has been detected." The input is the abnormality detection result and the emotion analysis result, and the output is a notification message.

[0501] Step 6:

[0502] The device receives the notification from the server and notifies the user by voice. For example, it plays a voice message such as "Are you OK? A fall has been detected." The input is the notification message, and the output is the voice notification.

[0503] Step 7:

[0504] The user responds to the notification from the terminal. For example, they reply, "It's okay." The input is the user's voice response, and the output is the confirmation result from the terminal.

[0505] Step 8:

[0506] If the device does not receive a response from the user, it automatically makes an emergency call, for example by automatically dialing an emergency number and contacting emergency services. The input is the user's response (if there is no response), and the output is the emergency call.

[0507] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0508] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0509] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0510] [Second embodiment]

[0511] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0512] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0513] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0514] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0515] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0516] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0517] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0518] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0519] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0520] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0521] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0522] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0523] The present invention provides a system that collects image data and audio data, analyzes them to detect abnormalities, and when an abnormality is detected, issues a notification and, if necessary, makes an emergency call.

[0524] System Configuration

[0525] server

[0526] The server analyzes the collected data and determines whether there are any abnormalities. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image diagnostic technology and voice recognition technology.

[0527] Terminal

[0528] The device is installed in the user's home and collects images and sounds on a daily basis. It also receives notifications from the server and conveys them to the user. The device has a built-in camera, microphone, and speaker.

[0529] User

[0530] The user is an elderly person living alone. He interacts with the device during his normal daily life and follows instructions from the device when necessary.

[0531] Program processing

[0532] Server Processing

[0533] 1. Data Reception and Analysis

[0534] The server receives the image data and audio data sent from the terminal. For example, it receives the user's image at a fixed time every day and analyzes the audio data.

[0535] The server uses image diagnostic technology to analyze the user's posture, complexion, etc., and detects abnormalities. For example, if the user falls or looks pale, it will determine that there is an abnormality.

[0536] Using voice recognition technology, the collected voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[0537] 2. Identifying and notifying abnormalities

[0538] The server determines whether there is an abnormality based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device, such as a message saying "A fall has been detected."

[0539] 3. Data recording and analysis

[0540] All data and events are logged and analyzed over the long term, allowing users to understand their health and lifestyle patterns.

[0541] Terminal handling

[0542] 1. Monitoring and Data Transmission

[0543] The device periodically takes pictures and records audio. For example, it takes a picture of the user every morning at 7:00 and sends the data to a server.

[0544] The collected data is sent to a server in real time via Wi-Fi.

[0545] 2. Receiving notifications and notifying users

[0546] When an abnormality notification is received from the server, the device will notify the user by voice, for example, saying, "Are you OK? A fall has been detected."

[0547] If the user does not respond, the device will automatically make an emergency call, for example by automatically dialing 119 to contact emergency services.

[0548] 3. Proposals for maintaining health and improving lifestyle habits

[0549] The device periodically provides users with advice on maintaining their health and lifestyle habits, such as voice messages like "Eat some vegetables today" or "Get outside and do some light exercise."

[0550] Specific examples

[0551] Scenario: User falls

[0552] 1. The user falls in the middle of the night.

[0553] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[0554] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0555] 4. The server immediately sends a notification of the fall accident to the device.

[0556] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[0557] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[0558] In this way, this system can ensure safety and security for elderly people living alone.

[0559] The processing flow will be explained below.

[0560] Server Processing

[0561] Step 1: Receiving data

[0562] The server receives image data and audio data sent from the device. For example, it receives a photo of the user's face, an image of the room, and recorded audio every hour.

[0563] Step 2: Image analysis

[0564] The server analyzes the received image data using image diagnostic technology, for example, analyzing the user's posture and facial expression to determine whether they are normal.

[0565] Step 3: Audio analysis

[0566] Using voice recognition technology, the received voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding like someone is in pain). For example, it checks whether the data contains the sound of a fall.

[0567] Step 4: Identifying anomalies

[0568] The server determines whether or not there is an abnormality based on the results of image and audio analysis. If an abnormality is detected, it proceeds to the next processing step.

[0569] Step 5: Generate notifications

[0570] If an abnormality is detected, the server generates and sends a notification to the device, such as a message saying "A fall has been detected."

[0571] Step 6: Data recording

[0572] The server records all events and analysis results. For example, it saves a log such as "January 10th, fall detected, notification sent."

[0573] Terminal handling

[0574] Step 1: Start monitoring

[0575] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[0576] Step 2: Send data

[0577] The device transmits the collected image and audio data to the server in real time.

[0578] Step 3: Receive notifications

[0579] Receive notifications from the server, for example, a message saying "A fall has been detected."

[0580] Step 4: Notify users

[0581] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[0582] Step 5: Check the response

[0583] Check the user's response, for example, "It's okay."

[0584] Step 6: Call Emergency Services

[0585] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[0586] Step 7: Regular advice

[0587] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[0588] User Behavior

[0589] Step 1: Living a normal life

[0590] The user goes about their daily life as normal.

[0591] Step 2: Voice interaction

[0592] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[0593] Step 3: Responding to emergencies

[0594] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[0595] Step 4: Stay healthy and take advice

[0596] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[0597] The above is the specific processing flow of the system. This system can provide safety and security to elderly people living alone.

[0598] Example 1

[0599] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] Elderly people living alone need to be able to quickly detect abnormalities in their daily lives, such as falls or sudden illness, and respond appropriately. However, conventional monitoring systems have a high risk of delaying emergency response due to low accuracy in detecting abnormalities and delayed notification. Furthermore, preventive health management is difficult due to a lack of support for understanding long-term health conditions and improving lifestyle habits. The purpose of this invention is to provide a system that solves these problems and improves the safety and security of elderly people living alone.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0602] In this invention, the server includes a means for receiving image data and audio data, a means for analyzing the image data and audio data to detect an abnormality, and a means for sending a notification when an abnormality is detected, thereby enabling the abnormality to be detected quickly and accurately and notified to the user in real time.

[0603] "Image data" is digital data that includes visual information captured by a camera or the like.

[0604] "Audio data" is digital data that includes sound information collected by a microphone or the like.

[0605] "Means for detecting anomalies" refers to technologies and methods for analyzing image and audio data to detect events that deviate from the user's normal behavioral patterns or states (e.g., falls, distressed voices, etc.).

[0606] "Means for sending notifications" refers to techniques or methods for communicating information about an abnormality to other devices or users when that information is detected.

[0607] "Means for notifying the user" refers to techniques or methods for notifying the user of the received notification, including voice messages and display screens.

[0608] "Means for making an emergency call" refers to the technology or method for contacting emergency services (e.g., ambulance, police) if the user does not respond to the abnormality notification.

[0609] "Means for recording data and conducting long-term analysis" refers to techniques and methods for storing collected data and analyzing a user's health status and lifestyle patterns over time.

[0610] "Image processing technology" refers to technology for analyzing digital images, and includes methods such as facial recognition, pose estimation, and object detection.

[0611] "Speech analysis technology" refers to technology for analyzing voice data, and includes methods such as voice recognition, acoustic feature extraction, and abnormal sound detection.

[0612] This invention is a system that collects image and voice data, analyzes them to detect abnormalities, notifies users when an abnormality is detected, and sends an emergency call if necessary. This system ensures the safety and security of elderly people living alone.

[0613] System Configuration

[0614] server

[0615] The server analyzes the collected data and determines whether there are any abnormalities. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image processing and voice analysis technologies. Specifically, the server uses an image processing library such as OpenCV to analyze the user's posture and facial expression, and uses voice recognition technology such as Google Cloud Speech-to-Text to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[0616] Terminal

[0617] The device is installed in the user's home and collects images and audio on a daily basis. The device has a built-in camera, microphone, and speaker, which are used to collect data and send it to a server. The device communicates with the server via Wi-Fi and transmits the collected data in real time. It also receives notifications from the server and relays them to the user. If an abnormality is detected, the device notifies the user by voice, and if the user does not respond, it makes an emergency call.

[0618] User

[0619] The user is an elderly person living alone. They interact with the device as they go about their daily lives and follow instructions from the device when necessary. For example, the device may provide advice such as "Go outside and do some light exercise."

[0620] Specific examples of data reception and analysis

[0621] For example, if a user falls in the middle of the night, the device will detect an abnormal sound (the sound of a fall). When that data is sent to the server, the server analyzes the received audio data and determines that it is likely a fall accident. The server immediately sends a notification of the fall accident to the device. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response. If the user does not respond, the device will automatically dial "119" to contact emergency services.

[0622] Example prompts for generative AI models

[0623] Please explain how the server and device work together to respond when the user falls, including detailed processing steps and specific actions.

[0624] In this way, the present invention provides a system that uses a high-performance server and terminals to ensure safety and security for elderly people living alone.

[0625] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0626] Step 1:

[0627] The device periodically collects image and audio data using a camera and microphone. The input is a user's photo and surrounding audio, and the output is a digital image file and audio file. For example, a device may take a photo of the user and record their audio every morning at 7:00. Specifically, the device's camera takes a picture and the microphone starts recording.

[0628] Step 2:

[0629] The device sends the collected image and audio data to the server via Wi-Fi. The input is the image and audio files stored on the device, and the output is the data packets sent to the server. Specifically, the device checks for connection to the Wi-Fi network and transfers the files to the server using a data transmission protocol (e.g., FTP).

[0630] Step 3:

[0631] The server performs a consistency check on the image and audio data received from the terminal and begins analysis. The input is the received image and audio files, and the output is data ready for analysis. Specifically, the server records a data reception log and verifies the file format.

[0632] Step 4:

[0633] The server uses OpenCV to analyze image data and detect the user's posture and facial color. The input is an image file, and the output is a repository of analysis results. The server first performs face recognition and then runs a posture estimation algorithm, for example, by detecting contours to determine whether the user's posture is normal or abnormal.

[0634] Step 5:

[0635] The server uses Google Cloud Speech-to-Text to analyze audio data and detect abnormal sounds. The input is an audio file, and the output is the audio data converted into text and the results of abnormal sound detection. Specifically, the server extracts features from the audio waveform and detects, for example, sounds of falling or sounds of distress.

[0636] Step 6:

[0637] The server integrates the analysis results of the image data and audio data to determine whether or not an abnormality has occurred. The inputs are the image analysis results and audio analysis results, and the output is the abnormality detection judgment result. Specifically, the server executes the abnormality detection algorithm and makes a judgment such as "A fall has been detected."

[0638] Step 7:

[0639] The server sends a notification to the terminal when an abnormality is detected. The input is the result of the abnormality detection judgment, and the output is a notification message to the terminal. In concrete terms, the server generates the notification message and sends it to the terminal via the network.

[0640] Step 8:

[0641] The device receives the notification from the server and notifies the user by voice. The received notification message is the input, and the voice message is generated as the output. Specifically, the device calls out to the user through the speaker, saying, "Are you OK? A fall has been detected."

[0642] Step 9:

[0643] If the user does not respond, the device automatically makes an emergency call. The input is the information that the user did not respond, and the output is a call to emergency services. Specifically, the device automatically dials "119" and notifies emergency services of the accident situation.

[0644] (Application example 1)

[0645] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0646] In recent years, there has been a growing emphasis on strengthening home security and improving the safety of home dwellers, including the elderly and those living alone. However, conventional security systems often lack a rapid or efficient response when an abnormality occurs. A particular issue is the lack of real-time abnormality detection and emergency reporting. The present invention aims to solve these issues by providing a system that instantly detects abnormalities in the home, notifies the user, and responds quickly.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0648] In this invention, the server includes means for collecting image data and audio data, means for analyzing the image data and audio data to detect abnormalities, means for analyzing data from sensors installed in the home in real time, means for sending real-time notifications to a smartphone based on the analysis results, and means for storing the data for later analysis and recording, thereby enabling immediate detection of abnormal situations in the home and prompt notification or emergency calls.

[0649] "Image data" refers to visual information acquired using an optical sensor and recorded as digital data.

[0650] "Audio data" refers to acoustic information collected through an acoustic sensor such as a microphone and recorded as digital data.

[0651] A "sensor" is a device that detects a physical or chemical phenomenon and outputs it as an electrical signal.

[0652] "Analysis" is the process of calculation and evaluation used to process collected data and find specific patterns or anomalies.

[0653] An "anomaly" is an event or condition that deviates from the normal or expected pattern.

[0654] A "notification" is a message or signal that notifies the user that an abnormality has been detected.

[0655] "Emergency notification" is a means of automatically sending a warning to pre-defined contacts when an abnormality occurs and the user does not respond.

[0656] "Real time" refers to the time range in which processing and notification occur with almost no delay after an event occurs.

[0657] "Storage" refers to saving the data used in the analysis and its results for future reference or analysis.

[0658] A "smartphone" is a portable information terminal that has the form of a mobile phone but has advanced computing power and can use a variety of applications.

[0659] This invention is a security system for ensuring safety within the home, which collects and analyzes image and audio data to detect abnormalities, notifies the user if an abnormality is detected, and makes an emergency call if there is no response from the user. Furthermore, by analyzing data in real time, it is possible to quickly detect and notify abnormalities.

[0660] server

[0661] The server is equipped with a high-performance processor and large-capacity storage, and performs data analysis using image diagnostics and speech recognition technologies. Specifically, it uses Python and its library, OpenCV, to perform image analysis, TensorFlow to run machine learning models, and Google Cloud Speech-to-Text API to transcribe and analyze voice data.

[0662] The server uses the following hardware and software:

[0663] Hardware: High-performance processors, server storage

[0664] Software: Python, OpenCV, TensorFlow, Google Cloud Speech-to-Text API

[0665] Terminal

[0666] The devices installed in homes are equipped with built-in cameras and microphones and periodically collect image and audio data. This collected data is then sent to a server in real time via Wi-Fi, minimizing the delay before the data reaches the server, achieving high real-time performance.

[0667] User

[0668] If an abnormality is detected, the user will be notified on their smartphone. This notification is sent as a push notification using Firebase Cloud Messaging. If the user does not respond to the notification, an emergency SMS will be sent to preset emergency contacts using the Twilio API.

[0669] Specific examples

[0670] For example, if a user hears the sound of an intruder breaking a window into their home late at night, a sensor installed in the home will detect this abnormal sound and send the data to a server in real time. The server will analyze the audio data and determine that an abnormal sound (intrusion sound) has been detected. At this time, the server will immediately send a push notification to the user's smartphone saying, "An abnormal sound has been detected. Please check the safety of your home." If the user does not respond to the notification, the app will automatically send an SMS to the designated emergency contact.

[0671] Specific examples of software and hardware used

[0672] Hardware: Home cameras, microphones, servers with high-performance processors, and users' smartphones

[0673] Software: Python, OpenCV, TensorFlow, Google Cloud Speech-to-Text API, Firebase Cloud Messaging, Twilio API

[0674] Prompt Sentence Examples

[0675] Describe the design of a smart security app that analyzes and notifies users in real time if any abnormal sounds (screams, breaking glass, etc.) are detected in their home.

[0676] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0677] Step 1:

[0678] The device is installed in the home and periodically collects image and audio data using a camera and microphone. The camera takes pictures at scheduled intervals, and the microphone records continuously. The collected data is sent to a server in real time via Wi-Fi.

[0679] Input: Real-time image and audio data from cameras and microphones

[0680] Output: Image and audio data transmitted via Wi-Fi

[0681] Step 2:

[0682] The server receives the image data and audio data sent from the device. The received data is read into the Python program to analyze it.

[0683] Input: Image and audio data received via Wi-Fi

[0684] Output: Image and audio data loaded for analysis

[0685] Step 3:

[0686] The server performs image analysis. It processes the image data using the OpenCV library and determines whether there are any abnormalities (e.g., a person lying down, a suspicious person entering, etc.).

[0687] Input: Imported image data

[0688] Data processing: Image processing using OpenCV, such as face recognition and motion detection

[0689] Output: Analysis results regarding the presence or absence of abnormalities

[0690] Step 4:

[0691] The server performs voice analysis. It transcribes the voice data using the Google Cloud Speech-to-Text API and then analyzes it. If any abnormal sounds are detected (e.g., screaming, breaking glass), the details are recorded.

[0692] Input: Loaded audio data

[0693] Data processing: Speech transcription and analysis using Google Cloud Speech-to-Text API

[0694] Output: Analysis results for abnormal sounds

[0695] Step 5:

[0696] If the server detects an anomaly, it generates a real-time notification based on the analysis results and sends a push notification to the smartphone using Firebase Cloud Messaging.

[0697] Input: Analysis results regarding the presence or absence of abnormalities

[0698] Data processing: Notification message generation

[0699] Output: Push notification to the user's smartphone

[0700] Step 6:

[0701] The user receives a notification and checks it on their smartphone. They read the notification message and check whether there are any abnormalities. They follow instructions such as "Make sure you are safe."

[0702] Input: Push notification to smartphone

[0703] Output: User response (confirmation or no response)

[0704] Step 7:

[0705] The server monitors the response to the notification. If there is no response from the user within a certain time, it will execute emergency notification procedures and send an SMS to the emergency contact using the Twilio API.

[0706] Input: User response (no response)

[0707] Data processing: Emergency call message generation

[0708] Output: SMS notification to emergency contacts

[0709] Step 8:

[0710] The server stores the data, recording the received image data, audio data, and analysis results in a database for later analysis.

[0711] Input: Analysis results, original data

[0712] Data processing: Data recording

[0713] Output: Stored database entries

[0714] Through these steps, the present invention quickly detects abnormal situations within the home and provides appropriate notification and emergency response.

[0715] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0716] This invention provides a system that collects image and audio data, analyzes them to detect abnormalities, notifies users when an abnormality is detected, and makes emergency calls if necessary.Furthermore, by incorporating an emotion engine that recognizes the user's emotions and takes appropriate action based on those emotions, it is possible to provide a higher level of safety and security.

[0717] System Configuration

[0718] server

[0719] The server analyzes the collected data and determines whether there are any abnormalities. It also uses an emotion engine to recognize the user's emotional state. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image diagnosis technology, voice recognition technology, and emotion analysis technology.

[0720] Terminal

[0721] The device is installed in the user's home and collects images and sounds on a daily basis. It also receives notifications from the server and conveys them to the user. The device has a built-in camera, microphone, and speaker.

[0722] User

[0723] The user is an elderly person living alone. He interacts with the device during his normal daily life and follows instructions from the device when necessary.

[0724] Program processing

[0725] Server Processing

[0726] 1. Data Reception and Analysis

[0727] The server receives the image data and audio data sent from the terminal. For example, it receives the user's image at a fixed time every day and analyzes the audio data.

[0728] The server uses image diagnostic technology to analyze the user's posture, complexion, etc., and detects abnormalities. For example, if the user falls or looks pale, it will determine that there is an abnormality.

[0729] Using voice recognition technology, the collected voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[0730] 2. Emotion analysis

[0731] The emotion engine analyzes the user's emotional state from image and voice data, determining, for example, whether the user is feeling stressed or happy.

[0732] 3. Identifying and notifying abnormalities

[0733] The server determines whether or not there are any abnormalities based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device, generating a message such as "A fall has been detected" or "The user is in a stressful state."

[0734] 4. Data recording and analysis

[0735] The server records all data and events as logs and performs long-term analysis, which allows the system to understand not only the user's health status and lifestyle patterns, but also emotional fluctuations.

[0736] Terminal handling

[0737] 1. Start monitoring

[0738] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[0739] 2. Data Transmission

[0740] The device transmits the collected image and audio data to the server in real time.

[0741] 3. Receiving notifications

[0742] Receive notifications from the server, for example, a message saying "A fall has been detected."

[0743] 4. Notice to Users

[0744] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[0745] 5. Response Acknowledgment

[0746] Check the user's response, for example, "It's okay."

[0747] 6. Emergency call

[0748] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[0749] 7. Providing regular advice

[0750] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[0751] User Behavior

[0752] 1. Normal life

[0753] The user goes about their daily life as normal.

[0754] 2. Voice Dialogue

[0755] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[0756] 3. Response in the event of an emergency

[0757] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[0758] 4. Stay healthy and take advice

[0759] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[0760] Specific examples

[0761] Scenario 1: User falls

[0762] 1. The user falls in the middle of the night.

[0763] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[0764] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0765] 4. The server immediately sends a notification of the fall accident to the device.

[0766] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[0767] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[0768] Scenario 2: The user is stressed

[0769] 1. The user is feeling stressed during the day.

[0770] 2. The device sends the captured images and recorded audio to the server.

[0771] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[0772] 4. The server sends a notification to the device saying "The user is in a stressed state."

[0773] 5. The device will advise the user by voice, "Take a deep breath to relax."

[0774] In this way, this system not only ensures the safety and security of elderly people living alone, but also monitors their emotional state, allowing for more comprehensive care.

[0775] The processing flow will be explained below.

[0776] Server Processing

[0777] Step 1: Receiving data

[0778] The server receives image data and audio data sent from the device. For example, it receives a photo of the user's face, an image of the room, and recorded audio every hour.

[0779] Step 2: Image analysis

[0780] The server analyzes the received image data using image diagnostic technology, for example, analyzing the user's posture (whether they have fallen) and facial expression (whether they look distressed) to determine whether there are any abnormalities.

[0781] Step 3: Audio analysis

[0782] Using voice recognition technology, the received voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding like someone is in pain). For example, it checks whether the data contains the sound of a fall.

[0783] Step 4: Sentiment Analysis

[0784] The emotion engine analyzes the user's emotional state from image and audio data, determining, for example, whether the user is feeling stressed or happy.

[0785] Step 5: Identifying anomalies

[0786] The server comprehensively evaluates the results of image analysis, audio analysis, and emotion analysis to determine whether or not there is an abnormality. For example, if a fall is detected and the user sounds distressed, it will determine that there is an abnormality.

[0787] Step 6: Generate notifications

[0788] If an abnormality is detected, the server generates and sends a notification to the device, such as a message saying "A fall has been detected" or "The user is in a stressed state."

[0789] Step 7: Data recording

[0790] The server records all data and events as logs for long-term analysis. For example, it stores logs such as "January 10th, 12:00, fall detected, notification sent."

[0791] Terminal handling

[0792] Step 1: Start monitoring

[0793] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[0794] Step 2: Send data

[0795] The device transmits the collected image and audio data to the server in real time.

[0796] Step 3: Receive notifications

[0797] Receive notifications from the server, for example, a message saying "A fall has been detected."

[0798] Step 4: Notify users

[0799] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[0800] Step 5: Check the response

[0801] Check the user's response, for example, "It's okay."

[0802] Step 6: Call Emergency Services

[0803] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[0804] Step 7: Regular advice

[0805] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[0806] User Behavior

[0807] Step 1: Living a normal life

[0808] The user goes about their daily life as normal.

[0809] Step 2: Voice interaction

[0810] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[0811] Step 3: Responding to emergencies

[0812] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[0813] Step 4: Stay healthy and take advice

[0814] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[0815] Specific examples

[0816] Scenario 1: User falls

[0817] Step 1: The user falls during the night.

[0818] Step 2: The device detects an abnormal sound (falling sound) and sends the data to the server.

[0819] Step 3: The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0820] Step 4: The server immediately sends a notification of the fall accident to the device.

[0821] Step 5: The device asks the user, "Are you OK? Did you fall?" and waits for a response.

[0822] Step 6: If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[0823] Scenario 2: The user is stressed

[0824] Step 1: The user experiences stress during the day.

[0825] Step 2: The device sends the captured images and recorded audio to the server.

[0826] Step 3: The server analyzes the received data and the emotion engine determines that the user is feeling stressed.

[0827] Step 4: The server sends a notification to the device saying "The user is in a stressed state."

[0828] Step 5: The device will advise the user by voice, "Take a deep breath to relax."

[0829] In this way, this system not only ensures the safety and security of elderly people living alone, but also monitors their emotional state, allowing for more comprehensive care.

[0830] Example 2

[0831] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0832] Ensuring safety and security is extremely important for elderly people living alone. However, conventional anomaly detection systems only target physical abnormalities and lack the functionality to monitor the user's emotional state and respond to changes. As a result, it has been difficult to alleviate the psychological burden, loneliness, and stress felt by elderly people. Furthermore, emergency responses are often inappropriate, requiring rapid action. To solve these issues, a system incorporating advanced analysis technology and an emotion engine is needed.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0834] In this invention, the server includes means for analyzing image data and audio data to detect abnormalities, means for analyzing the user's emotional state from the image data and audio data, and means for sending a notification when an abnormality is detected. This makes it possible to monitor not only physical abnormalities but also the user's emotional state. Specifically, by using emotion analysis technology in addition to image diagnosis technology and voice recognition technology, more accurate abnormality detection and emotion recognition can be achieved. Furthermore, by periodically providing advice on maintaining health and improving lifestyle habits, it is possible to improve the quality of life of elderly people. This provides a system that allows elderly people living alone to continue living more safely and with peace of mind.

[0835] "Image data" is a digital representation of visual information acquired by a photographic device such as a camera.

[0836] "Audio data" refers to a digital representation of sound information captured by a sound collection device such as a microphone.

[0837] "Means for detecting abnormalities" refers to the technology and process of analyzing image and audio data to detect abnormal events that deviate from normal life patterns.

[0838] "Means for analyzing emotional state" refers to the technology and process for recognizing and determining a user's emotional state and changes in emotions based on image and audio data.

[0839] "Means for sending notifications" refers to the techniques and processes that appropriately convey information to a terminal or other device when an abnormality or change in emotional state is detected.

[0840] "Means for receiving notifications and notifying users" refers to the technology and process for receiving notification information sent from a server or center and notifying the user of it by voice or other output means.

[0841] "Means for making emergency calls" refers to the technology and processes that automatically contact emergency services (e.g., an ambulance) if the user does not respond.

[0842] "Means for providing advice on maintaining health and improving lifestyle habits" refers to technologies and processes that periodically provide appropriate improvement measures and advice based on the user's health condition and lifestyle habits.

[0843] "Image diagnostic technology" is a technology that analyzes collected image data and recognizes various patterns and features to make a diagnosis.

[0844] "Speech recognition technology" is a technology that analyzes collected voice data and recognizes the content and characteristics of the voice.

[0845] "Emotion analysis technology" is a technology that analyzes and judges a user's emotional state based on image and audio data.

[0846] This invention relates to a system that collects and analyzes image and audio data to detect abnormalities and, if necessary, sends an emergency call. The purpose of this invention is to provide safety and security, particularly for elderly people living alone.

[0847] System Configuration

[0848] server

[0849] Servers are equipped with powerful processors and large storage capacity. They primarily use the following technologies and software:

[0850] OpenCV and TensorFlow are used as image diagnostic technologies.

[0851] Google Cloud Speech-to-Text and Amazon Transcribe are used as voice recognition technologies.

[0852] IBM Watson Tone Analyzer is used as the sentiment analysis technology.

[0853] This allows the collected image and audio data to be analyzed to determine the user's posture, facial color, abnormal sounds, and emotional state.

[0854] Terminal

[0855] The devices are installed in elderly people's homes and collect data on a daily basis. Specifically, they include the following hardware:

[0856] Camera module (e.g. Raspberry Pi camera module)

[0857] microphone

[0858] speaker

[0859] The device uses Python scripts to operate this hardware, periodically collecting images and audio and sending them to the server.

[0860] User

[0861] The user is an elderly person living alone, and can live a normal daily life without any special operations. The user interacts with the device in response to voice prompts.

[0862] Example

[0863] Scenario 1: User falls

[0864] 1. The user falls in the middle of the night.

[0865] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[0866] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0867] 4. The server immediately sends a notification of the fall accident to the device.

[0868] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[0869] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[0870] Scenario 2: The user is stressed

[0871] 1. The user is feeling stressed during the day.

[0872] 2. The device sends the captured images and recorded audio to the server.

[0873] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[0874] 4. The server sends a notification to the device saying "The user is in a stressed state."

[0875] 5. The device will advise the user by voice, "Take a deep breath to relax."

[0876] Prompt Sentence Examples

[0877] The following prompts can be used to generate analysis using a generative AI model:

[0878] "Analyze images of users falling and detect abnormalities."

[0879] "Analyze audio data containing the sound of a fall and detect any abnormalities."

[0880] "Analyze whether the user is in a stressful state."

[0881] This invention provides a system that allows elderly people living alone to continue living more safely and securely. By using a specific combination of hardware and software, highly accurate anomaly detection and emotion recognition become possible.

[0882] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0883] Program processing steps

[0884] Server Processing Steps

[0885] Step 1: Receiving data

[0886] The server receives image data (e.g., JPEG images) and audio data (e.g., WAV audio files) sent from the device. It receives the image data and audio data as input and stores them in temporary storage as output. After this storage, it is ready for analysis.

[0887] Step 2: Image analysis

[0888] The server analyzes the received image data using image diagnosis technology (OpenCV or TensorFlow). Specifically, it uses a face detection algorithm to identify the position of the face, and then analyzes facial color and posture to detect abnormalities. It receives image data as input and obtains analysis results (detection information on abnormal areas) as output.

[0889] Step 3: Audio analysis

[0890] The server converts the received voice data into text using speech recognition technology (Google Cloud Speech-to-Text or Amazon Transcribe), and then detects abnormal sounds. Specifically, it uses a deep learning model to identify sounds of falling or voices that sound like they're in pain. It receives voice data as input and obtains text data and the results of abnormal sound detection as output.

[0891] Step 4: Sentiment Analysis

[0892] The server uses an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotional state from image and voice data. Specifically, it analyzes voice tone and changes in facial expressions to determine stress and happiness. It receives image and voice data as input and obtains the analysis results of the emotional state as output.

[0893] Step 5: Identifying and notifying abnormalities

[0894] The server determines whether or not there is an abnormality based on the analysis results, and if an abnormality is detected, sends a notification to the device. It receives the analysis results as input, generates a notification message (e.g., "A fall has been detected") as output, and sends it to the device.

[0895] Step 6: Record and analyze data

[0896] The server logs all data and events into a non-relational database (MongoDB or Cassandra) for long-term analysis. It receives analysis data and event information as input and stores log data as output.

[0897] Terminal processing steps

[0898] Step 1: Start monitoring

[0899] The device periodically takes pictures and records audio using the built-in camera and microphone. Specifically, it takes pictures and records audio at 7:00 a.m. and 5:00 p.m. every day. The input is a time trigger, and the output is image data and audio data.

[0900] Step 2: Send data

[0901] The terminal transmits the collected image data and audio data to the server in real time. It receives the image data and audio data as input and saves a log of the transmission completion as output.

[0902] Step 3: Receive notifications

[0903] The device receives notifications from the server in real time via WebSocket or Firebase Cloud Messaging. It receives notification messages from the server as input and prepares the notification content as output.

[0904] Step 4: Notify users

[0905] The device uses a speech synthesis engine (such as Google Text-to-Speech) to communicate the received notification to the user aloud. Specifically, it plays a message such as "Are you OK? A fall has been detected" from the speaker. The device receives the notification message as input and plays the audio message as output.

[0906] Step 5: Check the response

[0907] The device collects the user's response using a microphone and analyzes it using a speech recognition engine. Specifically, it converts responses such as "It's okay" into text and sends it to the server. It receives voice data as input, generates text data as output, and sends it to the server.

[0908] Step 6: Call Emergency Services

[0909] If the user does not respond, the device automatically makes an emergency call. Specifically, it uses a module to connect to a telephone line and automatically dials "119." It receives the response confirmation result as input and sends the content of the call (e.g., "The user did not respond, please call an ambulance") as output.

[0910] Step 7: Regular advice

[0911] The device periodically provides users with voice advice on maintaining their health and improving their lifestyle. Specifically, it plays messages such as "Eat your vegetables today" from the speaker. Time is the trigger input, and the voice message is played as the output.

[0912] User action steps

[0913] Step 1: Living a normal life

[0914] Users can continue their daily lives as usual, and no special operations are required.

[0915] Step 2: Voice interaction

[0916] It interacts with the user in response to a prompt from the device. For example, the device asks, "How are you today?" and the user responds, "I'm fine." It receives the question from the device as input and responds as output.

[0917] Step 3: Responding to emergencies

[0918] When a user receives an abnormality notification from the device, they respond by following the instructions. For example, when asked "Did you fall?", they respond "Yes, I fell." The device receives a question from the device as input and responds as output.

[0919] Step 4: Stay healthy and take advice

[0920] Accept suggestions from the device for maintaining health and improving lifestyle habits. For example, try to maintain your health by following the meal menu suggested by the device. Receive suggestions from the device as input and execute them as output.

[0921] (Application example 2)

[0922] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0923] Elderly people living alone need a system that can not only detect falls and changes in their health, but also monitor their emotional state and take appropriate action. However, while current systems can detect abnormalities and issue emergency notifications, they do not take action based on emotional state, which means that they are unable to respond adequately when elderly people feel stressed or anxious.

[0924] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0925] In this invention, the server includes means for collecting image data and voice data, means for analyzing the image data and voice data to detect abnormalities, means for sending a notification when an abnormality is detected, means for receiving the notification and notifying the user, means for making an emergency call when there is no response to the notification, and means for analyzing the emotional state of the user using an emotion engine and taking appropriate measures as necessary. This ensures the safety and security of elderly people living alone and enables appropriate measures to be taken according to their emotional state.

[0926] "Image data" is digital data of visual information acquired using a photographing device such as a camera.

[0927] "Audio data" refers to digital data of auditory information acquired using a recording device such as a microphone.

[0928] "Analysis" is the process of using algorithms and programs to evaluate acquired image and audio data and detect specific patterns or anomalies.

[0929] An "abnormality" refers to a phenomenon that differs from normal conditions or behavior and requires an emergency response.

[0930] "Notification" refers to a warning or information message sent to the user when an abnormality is detected.

[0931] An "emergency call" is an action that automatically contacts emergency services or relevant parties when there is no response from the user.

[0932] An "emotion engine" is a control device or program that analyzes image data and audio data to recognize the user's emotional state and take appropriate action.

[0933] "Advice on maintaining health and improving lifestyle habits" refers to specific guidance and suggestions to help users live a healthy life.

[0934] "Image diagnostic technology" is a technology that analyzes image data to evaluate a user's posture, facial expression, and health condition.

[0935] "Voice recognition technology" is a technology for analyzing voice data to evaluate the content of a user's speech and the acoustic conditions.

[0936] This invention provides a system that collects image and audio data, analyzes them to detect abnormalities, and sends notifications and emergency calls as necessary.Furthermore, by using an emotion engine to recognize the user's emotional state and take appropriate measures, it is possible to provide users with a high level of safety and security.

[0937] System Configuration

[0938] Hardware:

[0939] 1. Camera: A device installed in the home that periodically captures images. An example is a network-enabled camera.

[0940] 2. Microphone: A device that collects sounds within the home. A highly sensitive microphone is used.

[0941] 3. Server: A device with a high-performance processor and large-capacity storage for analyzing data. It is mainly provided by AWS (Amazon Web Services) or on-premise high-performance computers.

[0942] 4. Terminal: A device installed in the home that communicates with the server. This includes smartphone apps and dedicated devices.

[0943] software:

[0944] 1. Flask: A server-side web application framework that allows the server to receive and process data.

[0945] 2. OpenCV: A library for image processing. It is used to detect anomalies in images.

[0946] 3. TensorFlow: A machine learning library for image and audio analysis, used for anomaly detection and sentiment analysis.

[0947] Server Roles

[0948] The server receives image and audio data sent from the device, analyzes it, and determines whether there are any abnormalities or the user's emotional state. The server has the following functions:

[0949] Data reception and analysis: Image and voice data are received and analyzed in real time. Image diagnostic and voice recognition technologies are used to evaluate the user's posture, complexion, tone of voice, etc.

[0950] Anomaly detection and notification: If an anomaly is detected, a notification is generated and sent to the device immediately, for example, a message saying "Fall detected."

[0951] Emotion analysis: Using an emotion engine to analyze a user's emotional state, for example, determining whether they are feeling stressed or happy.

[0952] Device Role

[0953] The terminal is installed in the home and communicates with the server. It has the following functions:

[0954] Monitoring and data transmission: Take pictures and record audio periodically and transmit them to a server. For example, run these at 7am and 5pm every day.

[0955] Notification reception and user notification: Receives notifications from the server and notifies the user by voice or text. For example, an audio message saying "A fall has been detected" is played.

[0956] Emergency Calls: Automatically make an emergency call if the user does not respond, for example by automatically dialing an emergency number and contacting emergency services.

[0957] Specific examples

[0958] Scenario 1: User falls

[0959] 1. The user falls in the middle of the night.

[0960] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[0961] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[0962] 4. The server immediately sends a notification of the fall accident to the device.

[0963] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[0964] 6. If the user does not respond, the device automatically makes an emergency call. The device automatically dials the emergency number and contacts emergency services.

[0965] Scenario 2: The user is stressed

[0966] 1. The user is feeling stressed during the day.

[0967] 2. The device sends the captured images and recorded audio to the server.

[0968] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[0969] 4. The server sends a notification to the device saying "The user is in a stressed state."

[0970] 5. The device will advise the user by voice, "Take a deep breath to relax."

[0971] Example prompt for a generative AI model:

[0972] Take the user's image and audio data as input, analyze it for abnormalities and emotional state, notify the user if an abnormality is detected, and provide advice on relaxation if the user is in a stressful state.

[0973] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0974] Step 1:

[0975] The device periodically takes pictures and records audio. For example, it does this every morning at 7:00 and evening at 5:00. The captured image data and recorded audio data are the input to the device. The device collects this data and prepares it for transmission. The output is image data and audio data.

[0976] Step 2:

[0977] The terminal sends the collected image data and audio data to the server. The transmission is done in real time. The image data and audio data output from the terminal become the input to the server. The data is transferred via the Internet. The server receives this data. The output is the received image data and audio data.

[0978] Step 3:

[0979] The server analyzes the image and audio data it receives. For the image data, OpenCV is used to evaluate the user's posture and facial color. For the audio data, TensorFlow is used to analyze abnormal sounds (such as the sound of a fall or a voice sounding distressed). The input is image data and audio data, and the output is the anomaly detection result.

[0980] Step 4:

[0981] The server uses an emotion engine to analyze the user's emotional state from the received image and audio data. For example, it determines whether the user is stressed or relaxed. The input is image and audio data, and the output is the emotion analysis result. The emotion engine uses a pre-trained generative AI model.

[0982] Step 5:

[0983] The server determines whether or not there is an abnormality based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device. For example, it generates a message saying "A fall has been detected." The input is the abnormality detection result and the emotion analysis result, and the output is a notification message.

[0984] Step 6:

[0985] The device receives the notification from the server and notifies the user by voice. For example, it plays a voice message such as "Are you OK? A fall has been detected." The input is the notification message, and the output is the voice notification.

[0986] Step 7:

[0987] The user responds to the notification from the terminal. For example, they reply, "It's okay." The input is the user's voice response, and the output is the confirmation result from the terminal.

[0988] Step 8:

[0989] If the device does not receive a response from the user, it automatically makes an emergency call, for example by automatically dialing an emergency number and contacting emergency services. The input is the user's response (if there is no response), and the output is the emergency call.

[0990] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0991] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0992] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0993] [Third embodiment]

[0994] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0995] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0996] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0997] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0998] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0999] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1000] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1001] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1002] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1003] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1004] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1005] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1006] The present invention provides a system that collects image data and audio data, analyzes them to detect abnormalities, and when an abnormality is detected, issues a notification and, if necessary, makes an emergency call.

[1007] System Configuration

[1008] server

[1009] The server analyzes the collected data and determines whether there are any abnormalities. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image diagnostic technology and voice recognition technology.

[1010] Terminal

[1011] The device is installed in the user's home and collects images and sounds on a daily basis. It also receives notifications from the server and conveys them to the user. The device has a built-in camera, microphone, and speaker.

[1012] User

[1013] The user is an elderly person living alone. He interacts with the device during his normal daily life and follows instructions from the device when necessary.

[1014] Program processing

[1015] Server Processing

[1016] 1. Data Reception and Analysis

[1017] The server receives the image data and audio data sent from the terminal. For example, it receives the user's image at a fixed time every day and analyzes the audio data.

[1018] The server uses image diagnostic technology to analyze the user's posture, complexion, etc., and detects abnormalities. For example, if the user falls or looks pale, it will determine that there is an abnormality.

[1019] Using voice recognition technology, the collected voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[1020] 2. Identifying and notifying abnormalities

[1021] The server determines whether there is an abnormality based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device, such as a message saying "A fall has been detected."

[1022] 3. Data recording and analysis

[1023] All data and events are logged and analyzed over the long term, allowing users to understand their health and lifestyle patterns.

[1024] Terminal handling

[1025] 1. Monitoring and Data Transmission

[1026] The device periodically takes pictures and records audio. For example, it takes a picture of the user every morning at 7:00 and sends the data to a server.

[1027] The collected data is sent to a server in real time via Wi-Fi.

[1028] 2. Receiving notifications and notifying users

[1029] When an abnormality notification is received from the server, the device will notify the user by voice, for example, saying, "Are you OK? A fall has been detected."

[1030] If the user does not respond, the device will automatically make an emergency call, for example by automatically dialing 119 to contact emergency services.

[1031] 3. Proposals for maintaining health and improving lifestyle habits

[1032] The device periodically provides users with advice on maintaining their health and lifestyle habits, such as voice messages like "Eat some vegetables today" or "Get outside and do some light exercise."

[1033] Specific examples

[1034] Scenario: User falls

[1035] 1. The user falls in the middle of the night.

[1036] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[1037] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1038] 4. The server immediately sends a notification of the fall accident to the device.

[1039] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[1040] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[1041] In this way, this system can ensure safety and security for elderly people living alone.

[1042] The processing flow will be explained below.

[1043] Server Processing

[1044] Step 1: Receiving data

[1045] The server receives image data and audio data sent from the device. For example, it receives a photo of the user's face, an image of the room, and recorded audio every hour.

[1046] Step 2: Image analysis

[1047] The server analyzes the received image data using image diagnostic technology, for example, analyzing the user's posture and facial expression to determine whether they are normal.

[1048] Step 3: Audio analysis

[1049] Using voice recognition technology, the received voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding like someone is in pain). For example, it checks whether the data contains the sound of a fall.

[1050] Step 4: Identifying anomalies

[1051] The server determines whether or not there is an abnormality based on the results of image and audio analysis. If an abnormality is detected, it proceeds to the next processing step.

[1052] Step 5: Generate notifications

[1053] If an abnormality is detected, the server generates and sends a notification to the device, such as a message saying "A fall has been detected."

[1054] Step 6: Data recording

[1055] The server records all events and analysis results. For example, it saves a log such as "January 10th, fall detected, notification sent."

[1056] Terminal handling

[1057] Step 1: Start monitoring

[1058] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[1059] Step 2: Send data

[1060] The device transmits the collected image and audio data to the server in real time.

[1061] Step 3: Receive notifications

[1062] Receive notifications from the server, for example, a message saying "A fall has been detected."

[1063] Step 4: Notify users

[1064] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[1065] Step 5: Check the response

[1066] Check the user's response, for example, "It's okay."

[1067] Step 6: Call Emergency Services

[1068] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[1069] Step 7: Regular advice

[1070] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[1071] User Behavior

[1072] Step 1: Living a normal life

[1073] The user goes about their daily life as normal.

[1074] Step 2: Voice interaction

[1075] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[1076] Step 3: Responding to emergencies

[1077] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[1078] Step 4: Stay healthy and take advice

[1079] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[1080] The above is the specific processing flow of the system. This system can provide safety and security to elderly people living alone.

[1081] Example 1

[1082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1083] Elderly people living alone need to be able to quickly detect abnormalities in their daily lives, such as falls or sudden illness, and respond appropriately. However, conventional monitoring systems have a high risk of delaying emergency response due to low accuracy in detecting abnormalities and delayed notification. Furthermore, preventive health management is difficult due to a lack of support for understanding long-term health conditions and improving lifestyle habits. The purpose of this invention is to provide a system that solves these problems and improves the safety and security of elderly people living alone.

[1084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1085] In this invention, the server includes a means for receiving image data and audio data, a means for analyzing the image data and audio data to detect an abnormality, and a means for sending a notification when an abnormality is detected, thereby enabling the abnormality to be detected quickly and accurately and notified to the user in real time.

[1086] "Image data" is digital data that includes visual information captured by a camera or the like.

[1087] "Audio data" is digital data that includes sound information collected by a microphone or the like.

[1088] "Means for detecting anomalies" refers to technologies and methods for analyzing image and audio data to detect events that deviate from the user's normal behavioral patterns or states (e.g., falls, distressed voices, etc.).

[1089] "Means for sending notifications" refers to techniques or methods for communicating information about an abnormality to other devices or users when that information is detected.

[1090] "Means for notifying the user" refers to techniques or methods for notifying the user of the received notification, including voice messages and display screens.

[1091] "Means for making an emergency call" refers to the technology or method for contacting emergency services (e.g., ambulance, police) if the user does not respond to the abnormality notification.

[1092] "Means for recording data and conducting long-term analysis" refers to techniques and methods for storing collected data and analyzing a user's health status and lifestyle patterns over time.

[1093] "Image processing technology" refers to technology for analyzing digital images, and includes methods such as facial recognition, pose estimation, and object detection.

[1094] "Speech analysis technology" refers to technology for analyzing voice data, and includes methods such as voice recognition, acoustic feature extraction, and abnormal sound detection.

[1095] This invention is a system that collects image and voice data, analyzes them to detect abnormalities, notifies users when an abnormality is detected, and sends an emergency call if necessary. This system ensures the safety and security of elderly people living alone.

[1096] System Configuration

[1097] server

[1098] The server analyzes the collected data and determines whether there are any abnormalities. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image processing and voice analysis technologies. Specifically, the server uses an image processing library such as OpenCV to analyze the user's posture and facial expression, and uses voice recognition technology such as Google Cloud Speech-to-Text to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[1099] Terminal

[1100] The device is installed in the user's home and collects images and audio on a daily basis. The device has a built-in camera, microphone, and speaker, which are used to collect data and send it to a server. The device communicates with the server via Wi-Fi and transmits the collected data in real time. It also receives notifications from the server and relays them to the user. If an abnormality is detected, the device notifies the user by voice, and if the user does not respond, it makes an emergency call.

[1101] User

[1102] The user is an elderly person living alone. They interact with the device as they go about their daily lives and follow instructions from the device when necessary. For example, the device may provide advice such as "Go outside and do some light exercise."

[1103] Specific examples of data reception and analysis

[1104] For example, if a user falls in the middle of the night, the device will detect an abnormal sound (the sound of a fall). When that data is sent to the server, the server analyzes the received audio data and determines that it is likely a fall accident. The server immediately sends a notification of the fall accident to the device. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response. If the user does not respond, the device will automatically dial "119" to contact emergency services.

[1105] Example prompts for generative AI models

[1106] Please explain how the server and device work together to respond when the user falls, including detailed processing steps and specific actions.

[1107] In this way, the present invention provides a system that uses a high-performance server and terminals to ensure safety and security for elderly people living alone.

[1108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1109] Step 1:

[1110] The device periodically collects image and audio data using a camera and microphone. The input is a user's photo and surrounding audio, and the output is a digital image file and audio file. For example, a device may take a photo of the user and record their audio every morning at 7:00. Specifically, the device's camera takes a picture and the microphone starts recording.

[1111] Step 2:

[1112] The device sends the collected image and audio data to the server via Wi-Fi. The input is the image and audio files stored on the device, and the output is the data packets sent to the server. Specifically, the device checks for connection to the Wi-Fi network and transfers the files to the server using a data transmission protocol (e.g., FTP).

[1113] Step 3:

[1114] The server performs a consistency check on the image and audio data received from the terminal and begins analysis. The input is the received image and audio files, and the output is data ready for analysis. Specifically, the server records a data reception log and verifies the file format.

[1115] Step 4:

[1116] The server uses OpenCV to analyze image data and detect the user's posture and facial color. The input is an image file, and the output is a repository of analysis results. The server first performs face recognition and then runs a posture estimation algorithm, for example, by detecting contours to determine whether the user's posture is normal or abnormal.

[1117] Step 5:

[1118] The server uses Google Cloud Speech-to-Text to analyze audio data and detect abnormal sounds. The input is an audio file, and the output is the audio data converted into text and the results of abnormal sound detection. Specifically, the server extracts features from the audio waveform and detects, for example, sounds of falling or sounds of distress.

[1119] Step 6:

[1120] The server integrates the analysis results of the image data and audio data to determine whether or not an abnormality has occurred. The inputs are the image analysis results and audio analysis results, and the output is the abnormality detection judgment result. Specifically, the server executes the abnormality detection algorithm and makes a judgment such as "A fall has been detected."

[1121] Step 7:

[1122] The server sends a notification to the terminal when an abnormality is detected. The input is the result of the abnormality detection judgment, and the output is a notification message to the terminal. In concrete terms, the server generates the notification message and sends it to the terminal via the network.

[1123] Step 8:

[1124] The device receives the notification from the server and notifies the user by voice. The received notification message is the input, and the voice message is generated as the output. Specifically, the device calls out to the user through the speaker, saying, "Are you OK? A fall has been detected."

[1125] Step 9:

[1126] If the user does not respond, the device automatically makes an emergency call. The input is the information that the user did not respond, and the output is a call to emergency services. Specifically, the device automatically dials "119" and notifies emergency services of the accident situation.

[1127] (Application example 1)

[1128] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1129] In recent years, there has been a growing emphasis on strengthening home security and improving the safety of home dwellers, including the elderly and those living alone. However, conventional security systems often lack a rapid or efficient response when an abnormality occurs. A particular issue is the lack of real-time abnormality detection and emergency reporting. The present invention aims to solve these issues by providing a system that instantly detects abnormalities in the home, notifies the user, and responds quickly.

[1130] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1131] In this invention, the server includes means for collecting image data and audio data, means for analyzing the image data and audio data to detect abnormalities, means for analyzing data from sensors installed in the home in real time, means for sending real-time notifications to a smartphone based on the analysis results, and means for storing the data for later analysis and recording, thereby enabling immediate detection of abnormal situations in the home and prompt notification or emergency calls.

[1132] "Image data" refers to visual information acquired using an optical sensor and recorded as digital data.

[1133] "Audio data" refers to acoustic information collected through an acoustic sensor such as a microphone and recorded as digital data.

[1134] A "sensor" is a device that detects a physical or chemical phenomenon and outputs it as an electrical signal.

[1135] "Analysis" is the process of calculation and evaluation used to process collected data and find specific patterns or anomalies.

[1136] An "anomaly" is an event or condition that deviates from the normal or expected pattern.

[1137] A "notification" is a message or signal that notifies the user that an abnormality has been detected.

[1138] "Emergency notification" is a means of automatically sending a warning to pre-defined contacts when an abnormality occurs and the user does not respond.

[1139] "Real time" refers to the time range in which processing and notification occur with almost no delay after an event occurs.

[1140] "Storage" refers to saving the data used in the analysis and its results for future reference or analysis.

[1141] A "smartphone" is a portable information terminal that has the form of a mobile phone but has advanced computing power and can use a variety of applications.

[1142] This invention is a security system for ensuring safety within the home, which collects and analyzes image and audio data to detect abnormalities, notifies the user if an abnormality is detected, and makes an emergency call if there is no response from the user. Furthermore, by analyzing data in real time, it is possible to quickly detect and notify abnormalities.

[1143] server

[1144] The server is equipped with a high-performance processor and large-capacity storage, and performs data analysis using image diagnostics and speech recognition technologies. Specifically, it uses Python and its library, OpenCV, to perform image analysis, TensorFlow to run machine learning models, and Google Cloud Speech-to-Text API to transcribe and analyze voice data.

[1145] The server uses the following hardware and software:

[1146] Hardware: High-performance processors, server storage

[1147] Software: Python, OpenCV, TensorFlow, Google Cloud Speech-to-Text API

[1148] Terminal

[1149] The devices installed in homes are equipped with built-in cameras and microphones and periodically collect image and audio data. This collected data is then sent to a server in real time via Wi-Fi, minimizing the delay before the data reaches the server, achieving high real-time performance.

[1150] User

[1151] If an abnormality is detected, the user will be notified on their smartphone. This notification is sent as a push notification using Firebase Cloud Messaging. If the user does not respond to the notification, an emergency SMS will be sent to preset emergency contacts using the Twilio API.

[1152] Specific examples

[1153] For example, if a user hears the sound of an intruder breaking a window into their home late at night, a sensor installed in the home will detect this abnormal sound and send the data to a server in real time. The server will analyze the audio data and determine that an abnormal sound (intrusion sound) has been detected. At this time, the server will immediately send a push notification to the user's smartphone saying, "An abnormal sound has been detected. Please check the safety of your home." If the user does not respond to the notification, the app will automatically send an SMS to the designated emergency contact.

[1154] Specific examples of software and hardware used

[1155] Hardware: Home cameras, microphones, servers with high-performance processors, and users' smartphones

[1156] Software: Python, OpenCV, TensorFlow, Google Cloud Speech-to-Text API, Firebase Cloud Messaging, Twilio API

[1157] Prompt Sentence Examples

[1158] Describe the design of a smart security app that analyzes and notifies users in real time if any abnormal sounds (screams, breaking glass, etc.) are detected in their home.

[1159] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1160] Step 1:

[1161] The device is installed in the home and periodically collects image and audio data using a camera and microphone. The camera takes pictures at scheduled intervals, and the microphone records continuously. The collected data is sent to a server in real time via Wi-Fi.

[1162] Input: Real-time image and audio data from cameras and microphones

[1163] Output: Image and audio data transmitted via Wi-Fi

[1164] Step 2:

[1165] The server receives the image data and audio data sent from the device. The received data is read into the Python program to analyze it.

[1166] Input: Image and audio data received via Wi-Fi

[1167] Output: Image and audio data loaded for analysis

[1168] Step 3:

[1169] The server performs image analysis. It processes the image data using the OpenCV library and determines whether there are any abnormalities (e.g., a person lying down, a suspicious person entering, etc.).

[1170] Input: Imported image data

[1171] Data processing: Image processing using OpenCV, such as face recognition and motion detection

[1172] Output: Analysis results regarding the presence or absence of abnormalities

[1173] Step 4:

[1174] The server performs voice analysis. It transcribes the voice data using the Google Cloud Speech-to-Text API and then analyzes it. If any abnormal sounds are detected (e.g., screaming, breaking glass), the details are recorded.

[1175] Input: Loaded audio data

[1176] Data processing: Speech transcription and analysis using Google Cloud Speech-to-Text API

[1177] Output: Analysis results for abnormal sounds

[1178] Step 5:

[1179] If the server detects an anomaly, it generates a real-time notification based on the analysis results and sends a push notification to the smartphone using Firebase Cloud Messaging.

[1180] Input: Analysis results regarding the presence or absence of abnormalities

[1181] Data processing: Notification message generation

[1182] Output: Push notification to the user's smartphone

[1183] Step 6:

[1184] The user receives a notification and checks it on their smartphone. They read the notification message and check whether there are any abnormalities. They follow instructions such as "Make sure you are safe."

[1185] Input: Push notification to smartphone

[1186] Output: User response (confirmation or no response)

[1187] Step 7:

[1188] The server monitors the response to the notification. If there is no response from the user within a certain time, it will execute emergency notification procedures and send an SMS to the emergency contact using the Twilio API.

[1189] Input: User response (no response)

[1190] Data processing: Emergency call message generation

[1191] Output: SMS notification to emergency contacts

[1192] Step 8:

[1193] The server stores the data, recording the received image data, audio data, and analysis results in a database for later analysis.

[1194] Input: Analysis results, original data

[1195] Data processing: Data recording

[1196] Output: Stored database entries

[1197] Through these steps, the present invention quickly detects abnormal situations within the home and provides appropriate notification and emergency response.

[1198] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1199] This invention provides a system that collects image and audio data, analyzes them to detect abnormalities, notifies users when an abnormality is detected, and makes emergency calls if necessary.Furthermore, by incorporating an emotion engine that recognizes the user's emotions and takes appropriate action based on those emotions, it is possible to provide a higher level of safety and security.

[1200] System Configuration

[1201] server

[1202] The server analyzes the collected data and determines whether there are any abnormalities. It also uses an emotion engine to recognize the user's emotional state. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image diagnosis technology, voice recognition technology, and emotion analysis technology.

[1203] Terminal

[1204] The device is installed in the user's home and collects images and sounds on a daily basis. It also receives notifications from the server and conveys them to the user. The device has a built-in camera, microphone, and speaker.

[1205] User

[1206] The user is an elderly person living alone. He interacts with the device during his normal daily life and follows instructions from the device when necessary.

[1207] Program processing

[1208] Server Processing

[1209] 1. Data Reception and Analysis

[1210] The server receives the image data and audio data sent from the terminal. For example, it receives the user's image at a fixed time every day and analyzes the audio data.

[1211] The server uses image diagnostic technology to analyze the user's posture, complexion, etc., and detects abnormalities. For example, if the user falls or looks pale, it will determine that there is an abnormality.

[1212] Using voice recognition technology, the collected voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[1213] 2. Emotion analysis

[1214] The emotion engine analyzes the user's emotional state from image and voice data, determining, for example, whether the user is feeling stressed or happy.

[1215] 3. Identifying and notifying abnormalities

[1216] The server determines whether or not there are any abnormalities based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device, generating a message such as "A fall has been detected" or "The user is in a stressful state."

[1217] 4. Data recording and analysis

[1218] The server records all data and events as logs and performs long-term analysis, which allows the system to understand not only the user's health status and lifestyle patterns, but also emotional fluctuations.

[1219] Terminal handling

[1220] 1. Start monitoring

[1221] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[1222] 2. Data Transmission

[1223] The device transmits the collected image and audio data to the server in real time.

[1224] 3. Receiving notifications

[1225] Receive notifications from the server, for example, a message saying "A fall has been detected."

[1226] 4. Notice to Users

[1227] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[1228] 5. Response Acknowledgment

[1229] Check the user's response, for example, "It's okay."

[1230] 6. Emergency call

[1231] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[1232] 7. Providing regular advice

[1233] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[1234] User Behavior

[1235] 1. Normal life

[1236] The user goes about their daily life as normal.

[1237] 2. Voice Dialogue

[1238] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[1239] 3. Response in the event of an emergency

[1240] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[1241] 4. Stay healthy and take advice

[1242] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[1243] Specific examples

[1244] Scenario 1: User falls

[1245] 1. The user falls in the middle of the night.

[1246] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[1247] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1248] 4. The server immediately sends a notification of the fall accident to the device.

[1249] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[1250] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[1251] Scenario 2: The user is stressed

[1252] 1. The user is feeling stressed during the day.

[1253] 2. The device sends the captured images and recorded audio to the server.

[1254] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[1255] 4. The server sends a notification to the device saying "The user is in a stressed state."

[1256] 5. The device will advise the user by voice, "Take a deep breath to relax."

[1257] In this way, this system not only ensures the safety and security of elderly people living alone, but also monitors their emotional state, allowing for more comprehensive care.

[1258] The processing flow will be explained below.

[1259] Server Processing

[1260] Step 1: Receiving data

[1261] The server receives image data and audio data sent from the device. For example, it receives a photo of the user's face, an image of the room, and recorded audio every hour.

[1262] Step 2: Image analysis

[1263] The server analyzes the received image data using image diagnostic technology, for example, analyzing the user's posture (whether they have fallen) and facial expression (whether they look distressed) to determine whether there are any abnormalities.

[1264] Step 3: Audio analysis

[1265] Using voice recognition technology, the received voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding like someone is in pain). For example, it checks whether the data contains the sound of a fall.

[1266] Step 4: Sentiment Analysis

[1267] The emotion engine analyzes the user's emotional state from image and audio data, determining, for example, whether the user is feeling stressed or happy.

[1268] Step 5: Identifying anomalies

[1269] The server comprehensively evaluates the results of image analysis, audio analysis, and emotion analysis to determine whether or not there is an abnormality. For example, if a fall is detected and the user sounds distressed, it will determine that there is an abnormality.

[1270] Step 6: Generate notifications

[1271] If an abnormality is detected, the server generates and sends a notification to the device, such as a message saying "A fall has been detected" or "The user is in a stressed state."

[1272] Step 7: Data recording

[1273] The server records all data and events as logs for long-term analysis. For example, it stores logs such as "January 10th, 12:00, fall detected, notification sent."

[1274] Terminal handling

[1275] Step 1: Start monitoring

[1276] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[1277] Step 2: Send data

[1278] The device transmits the collected image and audio data to the server in real time.

[1279] Step 3: Receive notifications

[1280] Receive notifications from the server, for example, a message saying "A fall has been detected."

[1281] Step 4: Notify users

[1282] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[1283] Step 5: Check the response

[1284] Check the user's response, for example, "It's okay."

[1285] Step 6: Call Emergency Services

[1286] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[1287] Step 7: Regular advice

[1288] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[1289] User Behavior

[1290] Step 1: Living a normal life

[1291] The user goes about their daily life as normal.

[1292] Step 2: Voice interaction

[1293] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[1294] Step 3: Responding to emergencies

[1295] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[1296] Step 4: Stay healthy and take advice

[1297] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[1298] Specific examples

[1299] Scenario 1: User falls

[1300] Step 1: The user falls during the night.

[1301] Step 2: The device detects an abnormal sound (falling sound) and sends the data to the server.

[1302] Step 3: The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1303] Step 4: The server immediately sends a notification of the fall accident to the device.

[1304] Step 5: The device asks the user, "Are you OK? Did you fall?" and waits for a response.

[1305] Step 6: If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[1306] Scenario 2: The user is stressed

[1307] Step 1: The user experiences stress during the day.

[1308] Step 2: The device sends the captured images and recorded audio to the server.

[1309] Step 3: The server analyzes the received data and the emotion engine determines that the user is feeling stressed.

[1310] Step 4: The server sends a notification to the device saying "The user is in a stressed state."

[1311] Step 5: The device will advise the user by voice, "Take a deep breath to relax."

[1312] In this way, this system not only ensures the safety and security of elderly people living alone, but also monitors their emotional state, allowing for more comprehensive care.

[1313] Example 2

[1314] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1315] Ensuring safety and security is extremely important for elderly people living alone. However, conventional anomaly detection systems only target physical abnormalities and lack the functionality to monitor the user's emotional state and respond to changes. As a result, it has been difficult to alleviate the psychological burden, loneliness, and stress felt by elderly people. Furthermore, emergency responses are often inappropriate, requiring rapid action. To solve these issues, a system incorporating advanced analysis technology and an emotion engine is needed.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1317] In this invention, the server includes means for analyzing image data and audio data to detect abnormalities, means for analyzing the user's emotional state from the image data and audio data, and means for sending a notification when an abnormality is detected. This makes it possible to monitor not only physical abnormalities but also the user's emotional state. Specifically, by using emotion analysis technology in addition to image diagnosis technology and voice recognition technology, more accurate abnormality detection and emotion recognition can be achieved. Furthermore, by periodically providing advice on maintaining health and improving lifestyle habits, it is possible to improve the quality of life of elderly people. This provides a system that allows elderly people living alone to continue living more safely and with peace of mind.

[1318] "Image data" is a digital representation of visual information acquired by a photographic device such as a camera.

[1319] "Audio data" refers to a digital representation of sound information captured by a sound collection device such as a microphone.

[1320] "Means for detecting abnormalities" refers to the technology and process of analyzing image and audio data to detect abnormal events that deviate from normal life patterns.

[1321] "Means for analyzing emotional state" refers to the technology and process for recognizing and determining a user's emotional state and changes in emotions based on image and audio data.

[1322] "Means for sending notifications" refers to the techniques and processes that appropriately convey information to a terminal or other device when an abnormality or change in emotional state is detected.

[1323] "Means for receiving notifications and notifying users" refers to the technology and process for receiving notification information sent from a server or center and notifying the user of it by voice or other output means.

[1324] "Means for making emergency calls" refers to the technology and processes that automatically contact emergency services (e.g., an ambulance) if the user does not respond.

[1325] "Means for providing advice on maintaining health and improving lifestyle habits" refers to technologies and processes that periodically provide appropriate improvement measures and advice based on the user's health condition and lifestyle habits.

[1326] "Image diagnostic technology" is a technology that analyzes collected image data and recognizes various patterns and features to make a diagnosis.

[1327] "Speech recognition technology" is a technology that analyzes collected voice data and recognizes the content and characteristics of the voice.

[1328] "Emotion analysis technology" is a technology that analyzes and judges a user's emotional state based on image and audio data.

[1329] This invention relates to a system that collects and analyzes image and audio data to detect abnormalities and, if necessary, sends an emergency call. The purpose of this invention is to provide safety and security, particularly for elderly people living alone.

[1330] System Configuration

[1331] server

[1332] Servers are equipped with powerful processors and large storage capacity. They primarily use the following technologies and software:

[1333] OpenCV and TensorFlow are used as image diagnostic technologies.

[1334] Google Cloud Speech-to-Text and Amazon Transcribe are used as voice recognition technologies.

[1335] IBM Watson Tone Analyzer is used as the sentiment analysis technology.

[1336] This allows the collected image and audio data to be analyzed to determine the user's posture, facial color, abnormal sounds, and emotional state.

[1337] Terminal

[1338] The devices are installed in elderly people's homes and collect data on a daily basis. Specifically, they include the following hardware:

[1339] Camera module (e.g. Raspberry Pi camera module)

[1340] microphone

[1341] speaker

[1342] The device uses Python scripts to operate this hardware, periodically collecting images and audio and sending them to the server.

[1343] User

[1344] The user is an elderly person living alone, and can live a normal daily life without any special operations. The user interacts with the device in response to voice prompts.

[1345] Example

[1346] Scenario 1: User falls

[1347] 1. The user falls in the middle of the night.

[1348] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[1349] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1350] 4. The server immediately sends a notification of the fall accident to the device.

[1351] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[1352] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[1353] Scenario 2: The user is stressed

[1354] 1. The user is feeling stressed during the day.

[1355] 2. The device sends the captured images and recorded audio to the server.

[1356] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[1357] 4. The server sends a notification to the device saying "The user is in a stressed state."

[1358] 5. The device will advise the user by voice, "Take a deep breath to relax."

[1359] Prompt Sentence Examples

[1360] The following prompts can be used to generate analysis using a generative AI model:

[1361] "Analyze images of users falling and detect abnormalities."

[1362] "Analyze audio data containing the sound of a fall and detect any abnormalities."

[1363] "Analyze whether the user is in a stressful state."

[1364] This invention provides a system that allows elderly people living alone to continue living more safely and securely. By using a specific combination of hardware and software, highly accurate anomaly detection and emotion recognition become possible.

[1365] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1366] Program processing steps

[1367] Server Processing Steps

[1368] Step 1: Receiving data

[1369] The server receives image data (e.g., JPEG images) and audio data (e.g., WAV audio files) sent from the device. It receives the image data and audio data as input and stores them in temporary storage as output. After this storage, it is ready for analysis.

[1370] Step 2: Image analysis

[1371] The server analyzes the received image data using image diagnosis technology (OpenCV or TensorFlow). Specifically, it uses a face detection algorithm to identify the position of the face, and then analyzes facial color and posture to detect abnormalities. It receives image data as input and obtains analysis results (detection information on abnormal areas) as output.

[1372] Step 3: Audio analysis

[1373] The server converts the received voice data into text using speech recognition technology (Google Cloud Speech-to-Text or Amazon Transcribe), and then detects abnormal sounds. Specifically, it uses a deep learning model to identify sounds of falling or voices that sound like they're in pain. It receives voice data as input and obtains text data and the results of abnormal sound detection as output.

[1374] Step 4: Sentiment Analysis

[1375] The server uses an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotional state from image and voice data. Specifically, it analyzes voice tone and changes in facial expressions to determine stress and happiness. It receives image and voice data as input and obtains the analysis results of the emotional state as output.

[1376] Step 5: Identifying and notifying abnormalities

[1377] The server determines whether or not there is an abnormality based on the analysis results, and if an abnormality is detected, sends a notification to the device. It receives the analysis results as input, generates a notification message (e.g., "A fall has been detected") as output, and sends it to the device.

[1378] Step 6: Record and analyze data

[1379] The server logs all data and events into a non-relational database (MongoDB or Cassandra) for long-term analysis. It receives analysis data and event information as input and stores log data as output.

[1380] Terminal processing steps

[1381] Step 1: Start monitoring

[1382] The device periodically takes pictures and records audio using the built-in camera and microphone. Specifically, it takes pictures and records audio at 7:00 a.m. and 5:00 p.m. every day. The input is a time trigger, and the output is image data and audio data.

[1383] Step 2: Send data

[1384] The terminal transmits the collected image data and audio data to the server in real time. It receives the image data and audio data as input and saves a log of the transmission completion as output.

[1385] Step 3: Receive notifications

[1386] The device receives notifications from the server in real time via WebSocket or Firebase Cloud Messaging. It receives notification messages from the server as input and prepares the notification content as output.

[1387] Step 4: Notify users

[1388] The device uses a speech synthesis engine (such as Google Text-to-Speech) to communicate the received notification to the user aloud. Specifically, it plays a message such as "Are you OK? A fall has been detected" from the speaker. The device receives the notification message as input and plays the audio message as output.

[1389] Step 5: Check the response

[1390] The device collects the user's response using a microphone and analyzes it using a speech recognition engine. Specifically, it converts responses such as "It's okay" into text and sends it to the server. It receives voice data as input, generates text data as output, and sends it to the server.

[1391] Step 6: Call Emergency Services

[1392] If the user does not respond, the device automatically makes an emergency call. Specifically, it uses a module to connect to a telephone line and automatically dials "119." It receives the response confirmation result as input and sends the content of the call (e.g., "The user did not respond, please call an ambulance") as output.

[1393] Step 7: Regular advice

[1394] The device periodically provides users with voice advice on maintaining their health and improving their lifestyle. Specifically, it plays messages such as "Eat your vegetables today" from the speaker. Time is the trigger input, and the voice message is played as the output.

[1395] User action steps

[1396] Step 1: Living a normal life

[1397] Users can continue their daily lives as usual, and no special operations are required.

[1398] Step 2: Voice interaction

[1399] It interacts with the user in response to a prompt from the device. For example, the device asks, "How are you today?" and the user responds, "I'm fine." It receives the question from the device as input and responds as output.

[1400] Step 3: Responding to emergencies

[1401] When a user receives an abnormality notification from the device, they respond by following the instructions. For example, when asked "Did you fall?", they respond "Yes, I fell." The device receives a question from the device as input and responds as output.

[1402] Step 4: Stay healthy and take advice

[1403] Accept suggestions from the device for maintaining health and improving lifestyle habits. For example, try to maintain your health by following the meal menu suggested by the device. Receive suggestions from the device as input and execute them as output.

[1404] (Application example 2)

[1405] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1406] Elderly people living alone need a system that can not only detect falls and changes in their health, but also monitor their emotional state and take appropriate action. However, while current systems can detect abnormalities and issue emergency notifications, they do not take action based on emotional state, which means that they are unable to respond adequately when elderly people feel stressed or anxious.

[1407] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1408] In this invention, the server includes means for collecting image data and voice data, means for analyzing the image data and voice data to detect abnormalities, means for sending a notification when an abnormality is detected, means for receiving the notification and notifying the user, means for making an emergency call when there is no response to the notification, and means for analyzing the emotional state of the user using an emotion engine and taking appropriate measures as necessary. This ensures the safety and security of elderly people living alone and enables appropriate measures to be taken according to their emotional state.

[1409] "Image data" is digital data of visual information acquired using a photographing device such as a camera.

[1410] "Audio data" refers to digital data of auditory information acquired using a recording device such as a microphone.

[1411] "Analysis" is the process of using algorithms and programs to evaluate acquired image and audio data and detect specific patterns or anomalies.

[1412] An "abnormality" refers to a phenomenon that differs from normal conditions or behavior and requires an emergency response.

[1413] "Notification" refers to a warning or information message sent to the user when an abnormality is detected.

[1414] An "emergency call" is an action that automatically contacts emergency services or relevant parties when there is no response from the user.

[1415] An "emotion engine" is a control device or program that analyzes image data and audio data to recognize the user's emotional state and take appropriate action.

[1416] "Advice on maintaining health and improving lifestyle habits" refers to specific guidance and suggestions to help users live a healthy life.

[1417] "Image diagnostic technology" is a technology that analyzes image data to evaluate a user's posture, facial expression, and health condition.

[1418] "Voice recognition technology" is a technology for analyzing voice data to evaluate the content of a user's speech and the acoustic conditions.

[1419] This invention provides a system that collects image and audio data, analyzes them to detect abnormalities, and sends notifications and emergency calls as necessary.Furthermore, by using an emotion engine to recognize the user's emotional state and take appropriate measures, it is possible to provide users with a high level of safety and security.

[1420] System Configuration

[1421] Hardware:

[1422] 1. Camera: A device installed in the home that periodically captures images. An example is a network-enabled camera.

[1423] 2. Microphone: A device that collects sounds within the home. A highly sensitive microphone is used.

[1424] 3. Server: A device with a high-performance processor and large-capacity storage for analyzing data. It is mainly provided by AWS (Amazon Web Services) or on-premise high-performance computers.

[1425] 4. Terminal: A device installed in the home that communicates with the server. This includes smartphone apps and dedicated devices.

[1426] software:

[1427] 1. Flask: A server-side web application framework that allows the server to receive and process data.

[1428] 2. OpenCV: A library for image processing. It is used to detect anomalies in images.

[1429] 3. TensorFlow: A machine learning library for image and audio analysis, used for anomaly detection and sentiment analysis.

[1430] Server Roles

[1431] The server receives image and audio data sent from the device, analyzes it, and determines whether there are any abnormalities or the user's emotional state. The server has the following functions:

[1432] Data reception and analysis: Image and voice data are received and analyzed in real time. Image diagnostic and voice recognition technologies are used to evaluate the user's posture, complexion, tone of voice, etc.

[1433] Anomaly detection and notification: If an anomaly is detected, a notification is generated and sent to the device immediately, for example, a message saying "Fall detected."

[1434] Emotion analysis: Using an emotion engine to analyze a user's emotional state, for example, determining whether they are feeling stressed or happy.

[1435] Device Role

[1436] The terminal is installed in the home and communicates with the server. It has the following functions:

[1437] Monitoring and data transmission: Take pictures and record audio periodically and transmit them to a server. For example, run these at 7am and 5pm every day.

[1438] Notification reception and user notification: Receives notifications from the server and notifies the user by voice or text. For example, an audio message saying "A fall has been detected" is played.

[1439] Emergency Calls: Automatically make an emergency call if the user does not respond, for example by automatically dialing an emergency number and contacting emergency services.

[1440] Specific examples

[1441] Scenario 1: User falls

[1442] 1. The user falls in the middle of the night.

[1443] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[1444] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1445] 4. The server immediately sends a notification of the fall accident to the device.

[1446] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[1447] 6. If the user does not respond, the device automatically makes an emergency call. The device automatically dials the emergency number and contacts emergency services.

[1448] Scenario 2: The user is stressed

[1449] 1. The user is feeling stressed during the day.

[1450] 2. The device sends the captured images and recorded audio to the server.

[1451] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[1452] 4. The server sends a notification to the device saying "The user is in a stressed state."

[1453] 5. The device will advise the user by voice, "Take a deep breath to relax."

[1454] Example prompt for a generative AI model:

[1455] Take the user's image and audio data as input, analyze it for abnormalities and emotional state, notify the user if an abnormality is detected, and provide advice on relaxation if the user is in a stressful state.

[1456] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1457] Step 1:

[1458] The device periodically takes pictures and records audio. For example, it does this every morning at 7:00 and evening at 5:00. The captured image data and recorded audio data are the input to the device. The device collects this data and prepares it for transmission. The output is image data and audio data.

[1459] Step 2:

[1460] The terminal sends the collected image data and audio data to the server. The transmission is done in real time. The image data and audio data output from the terminal become the input to the server. The data is transferred via the Internet. The server receives this data. The output is the received image data and audio data.

[1461] Step 3:

[1462] The server analyzes the image and audio data it receives. For the image data, OpenCV is used to evaluate the user's posture and facial color. For the audio data, TensorFlow is used to analyze abnormal sounds (such as the sound of a fall or a voice sounding distressed). The input is image data and audio data, and the output is the anomaly detection result.

[1463] Step 4:

[1464] The server uses an emotion engine to analyze the user's emotional state from the received image and audio data. For example, it determines whether the user is stressed or relaxed. The input is image and audio data, and the output is the emotion analysis result. The emotion engine uses a pre-trained generative AI model.

[1465] Step 5:

[1466] The server determines whether or not there is an abnormality based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device. For example, it generates a message saying "A fall has been detected." The input is the abnormality detection result and the emotion analysis result, and the output is a notification message.

[1467] Step 6:

[1468] The device receives the notification from the server and notifies the user by voice. For example, it plays a voice message such as "Are you OK? A fall has been detected." The input is the notification message, and the output is the voice notification.

[1469] Step 7:

[1470] The user responds to the notification from the terminal. For example, they reply, "It's okay." The input is the user's voice response, and the output is the confirmation result from the terminal.

[1471] Step 8:

[1472] If the device does not receive a response from the user, it automatically makes an emergency call, for example by automatically dialing an emergency number and contacting emergency services. The input is the user's response (if there is no response), and the output is the emergency call.

[1473] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1474] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1475] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1476] [Fourth embodiment]

[1477] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1478] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1479] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1480] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1481] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1482] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1483] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1484] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1485] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1486] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1487] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1488] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1489] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1490] The present invention provides a system that collects image data and audio data, analyzes them to detect abnormalities, and when an abnormality is detected, issues a notification and, if necessary, makes an emergency call.

[1491] System Configuration

[1492] server

[1493] The server analyzes the collected data and determines whether there are any abnormalities. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image diagnostic technology and voice recognition technology.

[1494] Terminal

[1495] The device is installed in the user's home and collects images and sounds on a daily basis. It also receives notifications from the server and conveys them to the user. The device has a built-in camera, microphone, and speaker.

[1496] User

[1497] The user is an elderly person living alone. He interacts with the device during his normal daily life and follows instructions from the device when necessary.

[1498] Program processing

[1499] Server Processing

[1500] 1. Data Reception and Analysis

[1501] The server receives the image data and audio data sent from the terminal. For example, it receives the user's image at a fixed time every day and analyzes the audio data.

[1502] The server uses image diagnostic technology to analyze the user's posture, complexion, etc., and detects abnormalities. For example, if the user falls or looks pale, it will determine that there is an abnormality.

[1503] Using voice recognition technology, the collected voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[1504] 2. Identifying and notifying abnormalities

[1505] The server determines whether there is an abnormality based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device, such as a message saying "A fall has been detected."

[1506] 3. Data recording and analysis

[1507] All data and events are logged and analyzed over the long term, allowing users to understand their health and lifestyle patterns.

[1508] Terminal handling

[1509] 1. Monitoring and Data Transmission

[1510] The device periodically takes pictures and records audio. For example, it takes a picture of the user every morning at 7:00 and sends the data to a server.

[1511] The collected data is sent to a server in real time via Wi-Fi.

[1512] 2. Receiving notifications and notifying users

[1513] When an abnormality notification is received from the server, the device will notify the user by voice, for example, saying, "Are you OK? A fall has been detected."

[1514] If the user does not respond, the device will automatically make an emergency call, for example by automatically dialing 119 to contact emergency services.

[1515] 3. Proposals for maintaining health and improving lifestyle habits

[1516] The device periodically provides users with advice on maintaining their health and lifestyle habits, such as voice messages like "Eat some vegetables today" or "Get outside and do some light exercise."

[1517] Specific examples

[1518] Scenario: User falls

[1519] 1. The user falls in the middle of the night.

[1520] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[1521] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1522] 4. The server immediately sends a notification of the fall accident to the device.

[1523] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[1524] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[1525] In this way, this system can ensure safety and security for elderly people living alone.

[1526] The processing flow will be explained below.

[1527] Server Processing

[1528] Step 1: Receiving data

[1529] The server receives image data and audio data sent from the device. For example, it receives a photo of the user's face, an image of the room, and recorded audio every hour.

[1530] Step 2: Image analysis

[1531] The server analyzes the received image data using image diagnostic technology, for example, analyzing the user's posture and facial expression to determine whether they are normal.

[1532] Step 3: Audio analysis

[1533] Using voice recognition technology, the received voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding like someone is in pain). For example, it checks whether the data contains the sound of a fall.

[1534] Step 4: Identifying anomalies

[1535] The server determines whether or not there is an abnormality based on the results of image and audio analysis. If an abnormality is detected, it proceeds to the next processing step.

[1536] Step 5: Generate notifications

[1537] If an abnormality is detected, the server generates and sends a notification to the device, such as a message saying "A fall has been detected."

[1538] Step 6: Data recording

[1539] The server records all events and analysis results. For example, it saves a log such as "January 10th, fall detected, notification sent."

[1540] Terminal handling

[1541] Step 1: Start monitoring

[1542] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[1543] Step 2: Send data

[1544] The device transmits the collected image and audio data to the server in real time.

[1545] Step 3: Receive notifications

[1546] Receive notifications from the server, for example, a message saying "A fall has been detected."

[1547] Step 4: Notify users

[1548] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[1549] Step 5: Check the response

[1550] Check the user's response, for example, "It's okay."

[1551] Step 6: Call Emergency Services

[1552] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[1553] Step 7: Regular advice

[1554] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[1555] User Behavior

[1556] Step 1: Living a normal life

[1557] The user goes about their daily life as normal.

[1558] Step 2: Voice interaction

[1559] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[1560] Step 3: Responding to emergencies

[1561] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[1562] Step 4: Stay healthy and take advice

[1563] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[1564] The above is the specific processing flow of the system. This system can provide safety and security to elderly people living alone.

[1565] Example 1

[1566] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1567] Elderly people living alone need to be able to quickly detect abnormalities in their daily lives, such as falls or sudden illness, and respond appropriately. However, conventional monitoring systems have a high risk of delaying emergency response due to low accuracy in detecting abnormalities and delayed notification. Furthermore, preventive health management is difficult due to a lack of support for understanding long-term health conditions and improving lifestyle habits. The purpose of this invention is to provide a system that solves these problems and improves the safety and security of elderly people living alone.

[1568] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1569] In this invention, the server includes a means for receiving image data and audio data, a means for analyzing the image data and audio data to detect an abnormality, and a means for sending a notification when an abnormality is detected, thereby enabling the abnormality to be detected quickly and accurately and notified to the user in real time.

[1570] "Image data" is digital data that includes visual information captured by a camera or the like.

[1571] "Audio data" is digital data that includes sound information collected by a microphone or the like.

[1572] "Means for detecting anomalies" refers to technologies and methods for analyzing image and audio data to detect events that deviate from the user's normal behavioral patterns or states (e.g., falls, distressed voices, etc.).

[1573] "Means for sending notifications" refers to techniques or methods for communicating information about an abnormality to other devices or users when that information is detected.

[1574] "Means for notifying the user" refers to techniques or methods for notifying the user of the received notification, including voice messages and display screens.

[1575] "Means for making an emergency call" refers to the technology or method for contacting emergency services (e.g., ambulance, police) if the user does not respond to the abnormality notification.

[1576] "Means for recording data and conducting long-term analysis" refers to techniques and methods for storing collected data and analyzing a user's health status and lifestyle patterns over time.

[1577] "Image processing technology" refers to technology for analyzing digital images, and includes methods such as facial recognition, pose estimation, and object detection.

[1578] "Speech analysis technology" refers to technology for analyzing voice data, and includes methods such as voice recognition, acoustic feature extraction, and abnormal sound detection.

[1579] This invention is a system that collects image and voice data, analyzes them to detect abnormalities, notifies users when an abnormality is detected, and sends an emergency call if necessary. This system ensures the safety and security of elderly people living alone.

[1580] System Configuration

[1581] server

[1582] The server analyzes the collected data and determines whether there are any abnormalities. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image processing and voice analysis technologies. Specifically, the server uses an image processing library such as OpenCV to analyze the user's posture and facial expression, and uses voice recognition technology such as Google Cloud Speech-to-Text to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[1583] Terminal

[1584] The device is installed in the user's home and collects images and audio on a daily basis. The device has a built-in camera, microphone, and speaker, which are used to collect data and send it to a server. The device communicates with the server via Wi-Fi and transmits the collected data in real time. It also receives notifications from the server and relays them to the user. If an abnormality is detected, the device notifies the user by voice, and if the user does not respond, it makes an emergency call.

[1585] User

[1586] The user is an elderly person living alone. They interact with the device as they go about their daily lives and follow instructions from the device when necessary. For example, the device may provide advice such as "Go outside and do some light exercise."

[1587] Specific examples of data reception and analysis

[1588] For example, if a user falls in the middle of the night, the device will detect an abnormal sound (the sound of a fall). When that data is sent to the server, the server analyzes the received audio data and determines that it is likely a fall accident. The server immediately sends a notification of the fall accident to the device. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response. If the user does not respond, the device will automatically dial "119" to contact emergency services.

[1589] Example prompts for generative AI models

[1590] Please explain how the server and device work together to respond when the user falls, including detailed processing steps and specific actions.

[1591] In this way, the present invention provides a system that uses a high-performance server and terminals to ensure safety and security for elderly people living alone.

[1592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1593] Step 1:

[1594] The device periodically collects image and audio data using a camera and microphone. The input is a user's photo and surrounding audio, and the output is a digital image file and audio file. For example, a device may take a photo of the user and record their audio every morning at 7:00. Specifically, the device's camera takes a picture and the microphone starts recording.

[1595] Step 2:

[1596] The device sends the collected image and audio data to the server via Wi-Fi. The input is the image and audio files stored on the device, and the output is the data packets sent to the server. Specifically, the device checks for connection to the Wi-Fi network and transfers the files to the server using a data transmission protocol (e.g., FTP).

[1597] Step 3:

[1598] The server performs a consistency check on the image and audio data received from the terminal and begins analysis. The input is the received image and audio files, and the output is data ready for analysis. Specifically, the server records a data reception log and verifies the file format.

[1599] Step 4:

[1600] The server uses OpenCV to analyze image data and detect the user's posture and facial color. The input is an image file, and the output is a repository of analysis results. The server first performs face recognition and then runs a posture estimation algorithm, for example, by detecting contours to determine whether the user's posture is normal or abnormal.

[1601] Step 5:

[1602] The server uses Google Cloud Speech-to-Text to analyze audio data and detect abnormal sounds. The input is an audio file, and the output is the audio data converted into text and the results of abnormal sound detection. Specifically, the server extracts features from the audio waveform and detects, for example, sounds of falling or sounds of distress.

[1603] Step 6:

[1604] The server integrates the analysis results of the image data and audio data to determine whether or not an abnormality has occurred. The inputs are the image analysis results and audio analysis results, and the output is the abnormality detection judgment result. Specifically, the server executes the abnormality detection algorithm and makes a judgment such as "A fall has been detected."

[1605] Step 7:

[1606] The server sends a notification to the terminal when an abnormality is detected. The input is the result of the abnormality detection judgment, and the output is a notification message to the terminal. In concrete terms, the server generates the notification message and sends it to the terminal via the network.

[1607] Step 8:

[1608] The device receives the notification from the server and notifies the user by voice. The received notification message is the input, and the voice message is generated as the output. Specifically, the device calls out to the user through the speaker, saying, "Are you OK? A fall has been detected."

[1609] Step 9:

[1610] If the user does not respond, the device automatically makes an emergency call. The input is the information that the user did not respond, and the output is a call to emergency services. Specifically, the device automatically dials "119" and notifies emergency services of the accident situation.

[1611] (Application example 1)

[1612] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1613] In recent years, there has been a growing emphasis on strengthening home security and improving the safety of home dwellers, including the elderly and those living alone. However, conventional security systems often lack a rapid or efficient response when an abnormality occurs. A particular issue is the lack of real-time abnormality detection and emergency reporting. The present invention aims to solve these issues by providing a system that instantly detects abnormalities in the home, notifies the user, and responds quickly.

[1614] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1615] In this invention, the server includes means for collecting image data and audio data, means for analyzing the image data and audio data to detect abnormalities, means for analyzing data from sensors installed in the home in real time, means for sending real-time notifications to a smartphone based on the analysis results, and means for storing the data for later analysis and recording, thereby enabling immediate detection of abnormal situations in the home and prompt notification or emergency calls.

[1616] "Image data" refers to visual information acquired using an optical sensor and recorded as digital data.

[1617] "Audio data" refers to acoustic information collected through an acoustic sensor such as a microphone and recorded as digital data.

[1618] A "sensor" is a device that detects a physical or chemical phenomenon and outputs it as an electrical signal.

[1619] "Analysis" is the process of calculation and evaluation used to process collected data and find specific patterns or anomalies.

[1620] An "anomaly" is an event or condition that deviates from the normal or expected pattern.

[1621] A "notification" is a message or signal that notifies the user that an abnormality has been detected.

[1622] "Emergency notification" is a means of automatically sending a warning to pre-defined contacts when an abnormality occurs and the user does not respond.

[1623] "Real time" refers to the time range in which processing and notification occur with almost no delay after an event occurs.

[1624] "Storage" refers to saving the data used in the analysis and its results for future reference or analysis.

[1625] A "smartphone" is a portable information terminal that has the form of a mobile phone but has advanced computing power and can use a variety of applications.

[1626] This invention is a security system for ensuring safety within the home, which collects and analyzes image and audio data to detect abnormalities, notifies the user if an abnormality is detected, and makes an emergency call if there is no response from the user. Furthermore, by analyzing data in real time, it is possible to quickly detect and notify abnormalities.

[1627] server

[1628] The server is equipped with a high-performance processor and large-capacity storage, and performs data analysis using image diagnostics and speech recognition technologies. Specifically, it uses Python and its library, OpenCV, to perform image analysis, TensorFlow to run machine learning models, and Google Cloud Speech-to-Text API to transcribe and analyze voice data.

[1629] The server uses the following hardware and software:

[1630] Hardware: High-performance processors, server storage

[1631] Software: Python, OpenCV, TensorFlow, Google Cloud Speech-to-Text API

[1632] Terminal

[1633] The devices installed in homes are equipped with built-in cameras and microphones and periodically collect image and audio data. This collected data is then sent to a server in real time via Wi-Fi, minimizing the delay before the data reaches the server, achieving high real-time performance.

[1634] User

[1635] If an abnormality is detected, the user will be notified on their smartphone. This notification is sent as a push notification using Firebase Cloud Messaging. If the user does not respond to the notification, an emergency SMS will be sent to preset emergency contacts using the Twilio API.

[1636] Specific examples

[1637] For example, if a user hears the sound of an intruder breaking a window into their home late at night, a sensor installed in the home will detect this abnormal sound and send the data to a server in real time. The server will analyze the audio data and determine that an abnormal sound (intrusion sound) has been detected. At this time, the server will immediately send a push notification to the user's smartphone saying, "An abnormal sound has been detected. Please check the safety of your home." If the user does not respond to the notification, the app will automatically send an SMS to the designated emergency contact.

[1638] Specific examples of software and hardware used

[1639] Hardware: Home cameras, microphones, servers with high-performance processors, and users' smartphones

[1640] Software: Python, OpenCV, TensorFlow, Google Cloud Speech-to-Text API, Firebase Cloud Messaging, Twilio API

[1641] Prompt Sentence Examples

[1642] Describe the design of a smart security app that analyzes and notifies users in real time if any abnormal sounds (screams, breaking glass, etc.) are detected in their home.

[1643] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1644] Step 1:

[1645] The device is installed in the home and periodically collects image and audio data using a camera and microphone. The camera takes pictures at scheduled intervals, and the microphone records continuously. The collected data is sent to a server in real time via Wi-Fi.

[1646] Input: Real-time image and audio data from cameras and microphones

[1647] Output: Image and audio data transmitted via Wi-Fi

[1648] Step 2:

[1649] The server receives the image data and audio data sent from the device. The received data is read into the Python program to analyze it.

[1650] Input: Image and audio data received via Wi-Fi

[1651] Output: Image and audio data loaded for analysis

[1652] Step 3:

[1653] The server performs image analysis. It processes the image data using the OpenCV library and determines whether there are any abnormalities (e.g., a person lying down, a suspicious person entering, etc.).

[1654] Input: Imported image data

[1655] Data processing: Image processing using OpenCV, such as face recognition and motion detection

[1656] Output: Analysis results regarding the presence or absence of abnormalities

[1657] Step 4:

[1658] The server performs voice analysis. It transcribes the voice data using the Google Cloud Speech-to-Text API and then analyzes it. If any abnormal sounds are detected (e.g., screaming, breaking glass), the details are recorded.

[1659] Input: Loaded audio data

[1660] Data processing: Speech transcription and analysis using Google Cloud Speech-to-Text API

[1661] Output: Analysis results for abnormal sounds

[1662] Step 5:

[1663] If the server detects an anomaly, it generates a real-time notification based on the analysis results and sends a push notification to the smartphone using Firebase Cloud Messaging.

[1664] Input: Analysis results regarding the presence or absence of abnormalities

[1665] Data processing: Notification message generation

[1666] Output: Push notification to the user's smartphone

[1667] Step 6:

[1668] The user receives a notification and checks it on their smartphone. They read the notification message and check whether there are any abnormalities. They follow instructions such as "Make sure you are safe."

[1669] Input: Push notification to smartphone

[1670] Output: User response (confirmation or no response)

[1671] Step 7:

[1672] The server monitors the response to the notification. If there is no response from the user within a certain time, it will execute emergency notification procedures and send an SMS to the emergency contact using the Twilio API.

[1673] Input: User response (no response)

[1674] Data processing: Emergency call message generation

[1675] Output: SMS notification to emergency contacts

[1676] Step 8:

[1677] The server stores the data, recording the received image data, audio data, and analysis results in a database for later analysis.

[1678] Input: Analysis results, original data

[1679] Data processing: Data recording

[1680] Output: Stored database entries

[1681] Through these steps, the present invention quickly detects abnormal situations within the home and provides appropriate notification and emergency response.

[1682] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1683] This invention provides a system that collects image and audio data, analyzes them to detect abnormalities, notifies users when an abnormality is detected, and makes emergency calls if necessary.Furthermore, by incorporating an emotion engine that recognizes the user's emotions and takes appropriate action based on those emotions, it is possible to provide a higher level of safety and security.

[1684] System Configuration

[1685] server

[1686] The server analyzes the collected data and determines whether there are any abnormalities. It also uses an emotion engine to recognize the user's emotional state. The server is equipped with a high-performance processor and large-capacity storage, and analyzes the data using image diagnosis technology, voice recognition technology, and emotion analysis technology.

[1687] Terminal

[1688] The device is installed in the user's home and collects images and sounds on a daily basis. It also receives notifications from the server and conveys them to the user. The device has a built-in camera, microphone, and speaker.

[1689] User

[1690] The user is an elderly person living alone. He interacts with the device during his normal daily life and follows instructions from the device when necessary.

[1691] Program processing

[1692] Server Processing

[1693] 1. Data Reception and Analysis

[1694] The server receives the image data and audio data sent from the terminal. For example, it receives the user's image at a fixed time every day and analyzes the audio data.

[1695] The server uses image diagnostic technology to analyze the user's posture, complexion, etc., and detects abnormalities. For example, if the user falls or looks pale, it will determine that there is an abnormality.

[1696] Using voice recognition technology, the collected voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding distressed).

[1697] 2. Emotion analysis

[1698] The emotion engine analyzes the user's emotional state from image and voice data, determining, for example, whether the user is feeling stressed or happy.

[1699] 3. Identifying and notifying abnormalities

[1700] The server determines whether or not there are any abnormalities based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device, generating a message such as "A fall has been detected" or "The user is in a stressful state."

[1701] 4. Data recording and analysis

[1702] The server records all data and events as logs and performs long-term analysis, which allows the system to understand not only the user's health status and lifestyle patterns, but also emotional fluctuations.

[1703] Terminal handling

[1704] 1. Start monitoring

[1705] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[1706] 2. Data Transmission

[1707] The device transmits the collected image and audio data to the server in real time.

[1708] 3. Receiving notifications

[1709] Receive notifications from the server, for example, a message saying "A fall has been detected."

[1710] 4. Notice to Users

[1711] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[1712] 5. Response Acknowledgment

[1713] Check the user's response, for example, "It's okay."

[1714] 6. Emergency call

[1715] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[1716] 7. Providing regular advice

[1717] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[1718] User Behavior

[1719] 1. Normal life

[1720] The user goes about their daily life as normal.

[1721] 2. Voice Dialogue

[1722] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[1723] 3. Response in the event of an emergency

[1724] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[1725] 4. Stay healthy and take advice

[1726] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[1727] Specific examples

[1728] Scenario 1: User falls

[1729] 1. The user falls in the middle of the night.

[1730] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[1731] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1732] 4. The server immediately sends a notification of the fall accident to the device.

[1733] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[1734] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[1735] Scenario 2: The user is stressed

[1736] 1. The user is feeling stressed during the day.

[1737] 2. The device sends the captured images and recorded audio to the server.

[1738] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[1739] 4. The server sends a notification to the device saying "The user is in a stressed state."

[1740] 5. The device will advise the user by voice, "Take a deep breath to relax."

[1741] In this way, this system not only ensures the safety and security of elderly people living alone, but also monitors their emotional state, allowing for more comprehensive care.

[1742] The processing flow will be explained below.

[1743] Server Processing

[1744] Step 1: Receiving data

[1745] The server receives image data and audio data sent from the device. For example, it receives a photo of the user's face, an image of the room, and recorded audio every hour.

[1746] Step 2: Image analysis

[1747] The server analyzes the received image data using image diagnostic technology, for example, analyzing the user's posture (whether they have fallen) and facial expression (whether they look distressed) to determine whether there are any abnormalities.

[1748] Step 3: Audio analysis

[1749] Using voice recognition technology, the received voice data is analyzed to detect abnormal sounds (such as the sound of a fall or a voice sounding like someone is in pain). For example, it checks whether the data contains the sound of a fall.

[1750] Step 4: Sentiment Analysis

[1751] The emotion engine analyzes the user's emotional state from image and audio data, determining, for example, whether the user is feeling stressed or happy.

[1752] Step 5: Identifying anomalies

[1753] The server comprehensively evaluates the results of image analysis, audio analysis, and emotion analysis to determine whether or not there is an abnormality. For example, if a fall is detected and the user sounds distressed, it will determine that there is an abnormality.

[1754] Step 6: Generate notifications

[1755] If an abnormality is detected, the server generates and sends a notification to the device, such as a message saying "A fall has been detected" or "The user is in a stressed state."

[1756] Step 7: Data recording

[1757] The server records all data and events as logs for long-term analysis. For example, it stores logs such as "January 10th, 12:00, fall detected, notification sent."

[1758] Terminal handling

[1759] Step 1: Start monitoring

[1760] The device takes pictures and records audio periodically, for example, at 7am and 5pm every day.

[1761] Step 2: Send data

[1762] The device transmits the collected image and audio data to the server in real time.

[1763] Step 3: Receive notifications

[1764] Receive notifications from the server, for example, a message saying "A fall has been detected."

[1765] Step 4: Notify users

[1766] The device will then notify the user of the received notification by voice, for example, saying, "Are you OK? A fall has been detected."

[1767] Step 5: Check the response

[1768] Check the user's response, for example, "It's okay."

[1769] Step 6: Call Emergency Services

[1770] If there is no response from the user, the device will automatically make an emergency call, for example by automatically dialing 119 and notifying emergency services.

[1771] Step 7: Regular advice

[1772] The device periodically provides users with advice on maintaining their health and improving their lifestyle, such as a message like "Eat some vegetables today."

[1773] User Behavior

[1774] Step 1: Living a normal life

[1775] The user goes about their daily life as normal.

[1776] Step 2: Voice interaction

[1777] The device responds to prompts from the user. For example, the device asks, "How are you today?" and the user responds, "I'm fine."

[1778] Step 3: Responding to emergencies

[1779] When an abnormality notification is received from the device, the device responds by following the instructions. For example, when asked "Did you fall?", the device responds "Yes, I fell."

[1780] Step 4: Stay healthy and take advice

[1781] Accept suggestions from the device for maintaining health and improving lifestyle habits, for example, by following the meal menu suggested by the device.

[1782] Specific examples

[1783] Scenario 1: User falls

[1784] Step 1: The user falls during the night.

[1785] Step 2: The device detects an abnormal sound (falling sound) and sends the data to the server.

[1786] Step 3: The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1787] Step 4: The server immediately sends a notification of the fall accident to the device.

[1788] Step 5: The device asks the user, "Are you OK? Did you fall?" and waits for a response.

[1789] Step 6: If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[1790] Scenario 2: The user is stressed

[1791] Step 1: The user experiences stress during the day.

[1792] Step 2: The device sends the captured images and recorded audio to the server.

[1793] Step 3: The server analyzes the received data and the emotion engine determines that the user is feeling stressed.

[1794] Step 4: The server sends a notification to the device saying "The user is in a stressed state."

[1795] Step 5: The device will advise the user by voice, "Take a deep breath to relax."

[1796] In this way, this system not only ensures the safety and security of elderly people living alone, but also monitors their emotional state, allowing for more comprehensive care.

[1797] Example 2

[1798] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1799] Ensuring safety and security is extremely important for elderly people living alone. However, conventional anomaly detection systems only target physical abnormalities and lack the functionality to monitor the user's emotional state and respond to changes. As a result, it has been difficult to alleviate the psychological burden, loneliness, and stress felt by elderly people. Furthermore, emergency responses are often inappropriate, requiring rapid action. To solve these issues, a system incorporating advanced analysis technology and an emotion engine is needed.

[1800] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1801] In this invention, the server includes means for analyzing image data and audio data to detect abnormalities, means for analyzing the user's emotional state from the image data and audio data, and means for sending a notification when an abnormality is detected. This makes it possible to monitor not only physical abnormalities but also the user's emotional state. Specifically, by using emotion analysis technology in addition to image diagnosis technology and voice recognition technology, more accurate abnormality detection and emotion recognition can be achieved. Furthermore, by periodically providing advice on maintaining health and improving lifestyle habits, it is possible to improve the quality of life of elderly people. This provides a system that allows elderly people living alone to continue living more safely and with peace of mind.

[1802] "Image data" is a digital representation of visual information acquired by a photographic device such as a camera.

[1803] "Audio data" refers to a digital representation of sound information captured by a sound collection device such as a microphone.

[1804] "Means for detecting abnormalities" refers to the technology and process of analyzing image and audio data to detect abnormal events that deviate from normal life patterns.

[1805] "Means for analyzing emotional state" refers to the technology and process for recognizing and determining a user's emotional state and changes in emotions based on image and audio data.

[1806] "Means for sending notifications" refers to the techniques and processes that appropriately convey information to a terminal or other device when an abnormality or change in emotional state is detected.

[1807] "Means for receiving notifications and notifying users" refers to the technology and process for receiving notification information sent from a server or center and notifying the user of it by voice or other output means.

[1808] "Means for making emergency calls" refers to the technology and processes that automatically contact emergency services (e.g., an ambulance) if the user does not respond.

[1809] "Means for providing advice on maintaining health and improving lifestyle habits" refers to technologies and processes that periodically provide appropriate improvement measures and advice based on the user's health condition and lifestyle habits.

[1810] "Image diagnostic technology" is a technology that analyzes collected image data and recognizes various patterns and features to make a diagnosis.

[1811] "Speech recognition technology" is a technology that analyzes collected voice data and recognizes the content and characteristics of the voice.

[1812] "Emotion analysis technology" is a technology that analyzes and judges a user's emotional state based on image and audio data.

[1813] This invention relates to a system that collects and analyzes image and audio data to detect abnormalities and, if necessary, sends an emergency call. The purpose of this invention is to provide safety and security, particularly for elderly people living alone.

[1814] System Configuration

[1815] server

[1816] Servers are equipped with powerful processors and large storage capacity. They primarily use the following technologies and software:

[1817] OpenCV and TensorFlow are used as image diagnostic technologies.

[1818] Google Cloud Speech-to-Text and Amazon Transcribe are used as voice recognition technologies.

[1819] IBM Watson Tone Analyzer is used as the sentiment analysis technology.

[1820] This allows the collected image and audio data to be analyzed to determine the user's posture, facial color, abnormal sounds, and emotional state.

[1821] Terminal

[1822] The devices are installed in elderly people's homes and collect data on a daily basis. Specifically, they include the following hardware:

[1823] Camera module (e.g. Raspberry Pi camera module)

[1824] microphone

[1825] speaker

[1826] The device uses Python scripts to operate this hardware, periodically collecting images and audio and sending them to the server.

[1827] User

[1828] The user is an elderly person living alone, and can live a normal daily life without any special operations. The user interacts with the device in response to voice prompts.

[1829] Example

[1830] Scenario 1: User falls

[1831] 1. The user falls in the middle of the night.

[1832] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[1833] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1834] 4. The server immediately sends a notification of the fall accident to the device.

[1835] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[1836] 6. If the user does not respond, the device automatically calls emergency services. The device automatically dials 119 and contacts emergency services.

[1837] Scenario 2: The user is stressed

[1838] 1. The user is feeling stressed during the day.

[1839] 2. The device sends the captured images and recorded audio to the server.

[1840] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[1841] 4. The server sends a notification to the device saying "The user is in a stressed state."

[1842] 5. The device will advise the user by voice, "Take a deep breath to relax."

[1843] Prompt Sentence Examples

[1844] The following prompts can be used to generate analysis using a generative AI model:

[1845] "Analyze images of users falling and detect abnormalities."

[1846] "Analyze audio data containing the sound of a fall and detect any abnormalities."

[1847] "Analyze whether the user is in a stressful state."

[1848] This invention provides a system that allows elderly people living alone to continue living more safely and securely. By using a specific combination of hardware and software, highly accurate anomaly detection and emotion recognition become possible.

[1849] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1850] Program processing steps

[1851] Server Processing Steps

[1852] Step 1: Receiving data

[1853] The server receives image data (e.g., JPEG images) and audio data (e.g., WAV audio files) sent from the device. It receives the image data and audio data as input and stores them in temporary storage as output. After this storage, it is ready for analysis.

[1854] Step 2: Image analysis

[1855] The server analyzes the received image data using image diagnosis technology (OpenCV or TensorFlow). Specifically, it uses a face detection algorithm to identify the position of the face, and then analyzes facial color and posture to detect abnormalities. It receives image data as input and obtains analysis results (detection information on abnormal areas) as output.

[1856] Step 3: Audio analysis

[1857] The server converts the received voice data into text using speech recognition technology (Google Cloud Speech-to-Text or Amazon Transcribe), and then detects abnormal sounds. Specifically, it uses a deep learning model to identify sounds of falling or voices that sound like they're in pain. It receives voice data as input and obtains text data and the results of abnormal sound detection as output.

[1858] Step 4: Sentiment Analysis

[1859] The server uses an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotional state from image and voice data. Specifically, it analyzes voice tone and changes in facial expressions to determine stress and happiness. It receives image and voice data as input and obtains the analysis results of the emotional state as output.

[1860] Step 5: Identifying and notifying abnormalities

[1861] The server determines whether or not there is an abnormality based on the analysis results, and if an abnormality is detected, sends a notification to the device. It receives the analysis results as input, generates a notification message (e.g., "A fall has been detected") as output, and sends it to the device.

[1862] Step 6: Record and analyze data

[1863] The server logs all data and events into a non-relational database (MongoDB or Cassandra) for long-term analysis. It receives analysis data and event information as input and stores log data as output.

[1864] Terminal processing steps

[1865] Step 1: Start monitoring

[1866] The device periodically takes pictures and records audio using the built-in camera and microphone. Specifically, it takes pictures and records audio at 7:00 a.m. and 5:00 p.m. every day. The input is a time trigger, and the output is image data and audio data.

[1867] Step 2: Send data

[1868] The terminal transmits the collected image data and audio data to the server in real time. It receives the image data and audio data as input and saves a log of the transmission completion as output.

[1869] Step 3: Receive notifications

[1870] The device receives notifications from the server in real time via WebSocket or Firebase Cloud Messaging. It receives notification messages from the server as input and prepares the notification content as output.

[1871] Step 4: Notify users

[1872] The device uses a speech synthesis engine (such as Google Text-to-Speech) to communicate the received notification to the user aloud. Specifically, it plays a message such as "Are you OK? A fall has been detected" from the speaker. The device receives the notification message as input and plays the audio message as output.

[1873] Step 5: Check the response

[1874] The device collects the user's response using a microphone and analyzes it using a speech recognition engine. Specifically, it converts responses such as "It's okay" into text and sends it to the server. It receives voice data as input, generates text data as output, and sends it to the server.

[1875] Step 6: Call Emergency Services

[1876] If the user does not respond, the device automatically makes an emergency call. Specifically, it uses a module to connect to a telephone line and automatically dials "119." It receives the response confirmation result as input and sends the content of the call (e.g., "The user did not respond, please call an ambulance") as output.

[1877] Step 7: Regular advice

[1878] The device periodically provides users with voice advice on maintaining their health and improving their lifestyle. Specifically, it plays messages such as "Eat your vegetables today" from the speaker. Time is the trigger input, and the voice message is played as the output.

[1879] User action steps

[1880] Step 1: Living a normal life

[1881] Users can continue their daily lives as usual, and no special operations are required.

[1882] Step 2: Voice interaction

[1883] It interacts with the user in response to a prompt from the device. For example, the device asks, "How are you today?" and the user responds, "I'm fine." It receives the question from the device as input and responds as output.

[1884] Step 3: Responding to emergencies

[1885] When a user receives an abnormality notification from the device, they respond by following the instructions. For example, when asked "Did you fall?", they respond "Yes, I fell." The device receives a question from the device as input and responds as output.

[1886] Step 4: Stay healthy and take advice

[1887] Accept suggestions from the device for maintaining health and improving lifestyle habits. For example, try to maintain your health by following the meal menu suggested by the device. Receive suggestions from the device as input and execute them as output.

[1888] (Application example 2)

[1889] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1890] Elderly people living alone need a system that can not only detect falls and changes in their health, but also monitor their emotional state and take appropriate action. However, while current systems can detect abnormalities and issue emergency notifications, they do not take action based on emotional state, which means that they are unable to respond adequately when elderly people feel stressed or anxious.

[1891] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1892] In this invention, the server includes means for collecting image data and voice data, means for analyzing the image data and voice data to detect abnormalities, means for sending a notification when an abnormality is detected, means for receiving the notification and notifying the user, means for making an emergency call when there is no response to the notification, and means for analyzing the emotional state of the user using an emotion engine and taking appropriate measures as necessary. This ensures the safety and security of elderly people living alone and enables appropriate measures to be taken according to their emotional state.

[1893] "Image data" is digital data of visual information acquired using a photographing device such as a camera.

[1894] "Audio data" refers to digital data of auditory information acquired using a recording device such as a microphone.

[1895] "Analysis" is the process of using algorithms and programs to evaluate acquired image and audio data and detect specific patterns or anomalies.

[1896] An "abnormality" refers to a phenomenon that differs from normal conditions or behavior and requires an emergency response.

[1897] "Notification" refers to a warning or information message sent to the user when an abnormality is detected.

[1898] An "emergency call" is an action that automatically contacts emergency services or relevant parties when there is no response from the user.

[1899] An "emotion engine" is a control device or program that analyzes image data and audio data to recognize the user's emotional state and take appropriate action.

[1900] "Advice on maintaining health and improving lifestyle habits" refers to specific guidance and suggestions to help users live a healthy life.

[1901] "Image diagnostic technology" is a technology that analyzes image data to evaluate a user's posture, facial expression, and health condition.

[1902] "Voice recognition technology" is a technology for analyzing voice data to evaluate the content of a user's speech and the acoustic conditions.

[1903] This invention provides a system that collects image and audio data, analyzes them to detect abnormalities, and sends notifications and emergency calls as necessary.Furthermore, by using an emotion engine to recognize the user's emotional state and take appropriate measures, it is possible to provide users with a high level of safety and security.

[1904] System Configuration

[1905] Hardware:

[1906] 1. Camera: A device installed in the home that periodically captures images. An example is a network-enabled camera.

[1907] 2. Microphone: A device that collects sounds within the home. A highly sensitive microphone is used.

[1908] 3. Server: A device with a high-performance processor and large-capacity storage for analyzing data. It is mainly provided by AWS (Amazon Web Services) or on-premise high-performance computers.

[1909] 4. Terminal: A device installed in the home that communicates with the server. This includes smartphone apps and dedicated devices.

[1910] software:

[1911] 1. Flask: A server-side web application framework that allows the server to receive and process data.

[1912] 2. OpenCV: A library for image processing. It is used to detect anomalies in images.

[1913] 3. TensorFlow: A machine learning library for image and audio analysis, used for anomaly detection and sentiment analysis.

[1914] Server Roles

[1915] The server receives image and audio data sent from the device, analyzes it, and determines whether there are any abnormalities or the user's emotional state. The server has the following functions:

[1916] Data reception and analysis: Image and voice data are received and analyzed in real time. Image diagnostic and voice recognition technologies are used to evaluate the user's posture, complexion, tone of voice, etc.

[1917] Anomaly detection and notification: If an anomaly is detected, a notification is generated and sent to the device immediately, for example, a message saying "Fall detected."

[1918] Emotion analysis: Using an emotion engine to analyze a user's emotional state, for example, determining whether they are feeling stressed or happy.

[1919] Device Role

[1920] The terminal is installed in the home and communicates with the server. It has the following functions:

[1921] Monitoring and data transmission: Take pictures and record audio periodically and transmit them to a server. For example, run these at 7am and 5pm every day.

[1922] Notification reception and user notification: Receives notifications from the server and notifies the user by voice or text. For example, an audio message saying "A fall has been detected" is played.

[1923] Emergency Calls: Automatically make an emergency call if the user does not respond, for example by automatically dialing an emergency number and contacting emergency services.

[1924] Specific examples

[1925] Scenario 1: User falls

[1926] 1. The user falls in the middle of the night.

[1927] 2. The device detects an abnormal sound (falling sound) and sends the data to the server.

[1928] 3. The server analyzes the received voice data and determines that there is a high possibility of a fall accident.

[1929] 4. The server immediately sends a notification of the fall accident to the device.

[1930] 5. The device will contact the user by voice, asking "Are you OK? Did you fall?" and wait for a response.

[1931] 6. If the user does not respond, the device automatically makes an emergency call. The device automatically dials the emergency number and contacts emergency services.

[1932] Scenario 2: The user is stressed

[1933] 1. The user is feeling stressed during the day.

[1934] 2. The device sends the captured images and recorded audio to the server.

[1935] 3. The server analyzes the received data, and the emotion engine determines that the user is feeling stressed.

[1936] 4. The server sends a notification to the device saying "The user is in a stressed state."

[1937] 5. The device will advise the user by voice, "Take a deep breath to relax."

[1938] Example prompt for a generative AI model:

[1939] Take the user's image and audio data as input, analyze it for abnormalities and emotional state, notify the user if an abnormality is detected, and provide advice on relaxation if the user is in a stressful state.

[1940] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1941] Step 1:

[1942] The device periodically takes pictures and records audio. For example, it does this every morning at 7:00 and evening at 5:00. The captured image data and recorded audio data are the input to the device. The device collects this data and prepares it for transmission. The output is image data and audio data.

[1943] Step 2:

[1944] The terminal sends the collected image data and audio data to the server. The transmission is done in real time. The image data and audio data output from the terminal become the input to the server. The data is transferred via the Internet. The server receives this data. The output is the received image data and audio data.

[1945] Step 3:

[1946] The server analyzes the image and audio data it receives. For the image data, OpenCV is used to evaluate the user's posture and facial color. For the audio data, TensorFlow is used to analyze abnormal sounds (such as the sound of a fall or a voice sounding distressed). The input is image data and audio data, and the output is the anomaly detection result.

[1947] Step 4:

[1948] The server uses an emotion engine to analyze the user's emotional state from the received image and audio data. For example, it determines whether the user is stressed or relaxed. The input is image and audio data, and the output is the emotion analysis result. The emotion engine uses a pre-trained generative AI model.

[1949] Step 5:

[1950] The server determines whether or not there is an abnormality based on the analysis results, and if an abnormality is detected, it immediately sends a notification to the device. For example, it generates a message saying "A fall has been detected." The input is the abnormality detection result and the emotion analysis result, and the output is a notification message.

[1951] Step 6:

[1952] The device receives the notification from the server and notifies the user by voice. For example, it plays a voice message such as "Are you OK? A fall has been detected." The input is the notification message, and the output is the voice notification.

[1953] Step 7:

[1954] The user responds to the notification from the terminal. For example, they reply, "It's okay." The input is the user's voice response, and the output is the confirmation result from the terminal.

[1955] Step 8:

[1956] If the device does not receive a response from the user, it automatically makes an emergency call, for example by automatically dialing an emergency number and contacting emergency services. The input is the user's response (if there is no response), and the output is the emergency call.

[1957] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1958] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1959] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1960] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1961] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1962] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1963] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1964] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1965] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1966] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1967] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1968] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1969] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1970] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1971] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1972] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1973] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1974] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1975] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1976] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1977] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1978] The following is further disclosed regarding the above embodiment.

[1979] (Claim 1)

[1980] means for collecting image data and audio data;

[1981] means for analyzing the image data and audio data to detect an abnormality;

[1982] means for sending a notification when an anomaly is detected;

[1983] means for receiving the notification and notifying the user;

[1984] means for making an emergency call if there is no response to said notification;

[1985] A system including:

[1986] (Claim 2)

[1987] 10. The system of claim 1, further comprising means for periodically providing the user with advice on maintaining health and improving lifestyle habits.

[1988] (Claim 3)

[1989] 10. The system of claim 1, wherein the analysis means uses image diagnostic technology and voice recognition technology.

[1990] (Claim 4)

[1991] 10. The system of claim 1, wherein the means for detecting anomalies identifies audio data that includes sounds of a fall or distress.

[1992] "Example 1"

[1993] (Claim 1)

[1994] means for collecting image data and audio data;

[1995] means for analyzing the image data and audio data to detect an abnormality;

[1996] means for sending a notification when an anomaly is detected;

[1997] means for receiving the notification and notifying the user;

[1998] means for making an emergency call if there is no response to said notification;

[1999] means for recording said data and for longitudinal analysis;

[2000] A system including:

[2001] (Claim 2)

[2002] 10. The system of claim 1, further comprising means for periodically providing the user with advice on maintaining health and improving lifestyle habits.

[2003] (Claim 3)

[2004] 10. The system of claim 1, wherein the analysis means uses image processing and audio analysis techniques.

[2005] "Application Example 1"

[2006] (Claim 1)

[2007] means for collecting image data and audio data;

[2008] means for analyzing the image data and audio data to detect an abnormality;

[2009] means for sending a notification when an anomaly is detected;

[2010] means for receiving the notification and notifying the user;

[2011] means for making an emergency call if there is no response to said notification;

[2012] A means for analyzing data from sensors installed in the home in real time;

[2013] A means to send real-time notifications to your smartphone based on the analysis results;

[2014] A means of storing the data for later analysis and documentation;

[2015] A system including:

[2016] (Claim 2)

[2017] 10. The system of claim 1, further comprising means for periodically providing the user with advice on maintaining health and improving lifestyle habits.

[2018] (Claim 3)

[2019] 10. The system of claim 1, wherein the analysis means uses image diagnostic technology and voice recognition technology.

[2020] "Example 2: Combining Emotion Engines"

[2021] (Claim 1)

[2022] means for collecting image data and audio data;

[2023] means for analyzing the image data and audio data to detect an abnormality;

[2024] means for analyzing the emotional state of a user from the image data and voice data;

[2025] means for sending a notification when an anomaly is detected;

[2026] means for receiving the notification and notifying the user;

[2027] means for making an emergency call if there is no response to said notification;

[2028] A system including:

[2029] (Claim 2)

[2030] 10. The system of claim 1, further comprising means for periodically providing the user with advice on maintaining health and improving lifestyle habits.

[2031] (Claim 3)

[2032] The system of claim 1, wherein the analysis means uses image diagnosis technology, voice recognition technology, and emotion analysis technology.

[2033] "Application example 2 when combining emotion engines"

[2034] (Claim 1)

[2035] means for collecting image data and audio data;

[2036] means for analyzing the image data and audio data to detect an abnormality;

[2037] means for sending a notification when an anomaly is detected;

[2038] means for receiving the notification and notifying the user;

[2039] means for making an emergency call if there is no response to said notification;

[2040] A means for analyzing the user's emotional state using an emotion engine and taking appropriate measures as needed;

[2041] A system including:

[2042] (Claim 2)

[2043] 10. The system of claim 1, further comprising means for periodically providing the user with advice on maintaining health and improving lifestyle habits.

[2044] (Claim 3)

[2045] 10. The system of claim 1, wherein the analysis means uses image diagnostic technology and voice recognition technology. [Explanation of symbols]

[2046] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting image data and audio data; means for analyzing the image data and audio data to detect an abnormality; means for sending a notification when an anomaly is detected; means for receiving the notification and notifying the user; means for making an emergency call if there is no response to said notification; A system including:

2. The system according to claim 1 , further comprising means for periodically providing the user with advice on maintaining health and improving lifestyle habits.

3. The system of claim 1 , wherein the analysis means uses imaging diagnostic techniques and voice recognition techniques.

4. The system of claim 1 , wherein the means for detecting anomalies identifies audio data containing sounds of a fall or distress.

Images