System

Abstract

A system is provided.SOLUTION: Means for collecting electric power usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting a web camera image in real time, and means for analyzing the electric power usage information, the water usage information, the gas usage information, and the web camera image, the system includes a means for detecting an abnormality, a means for issuing a warning when the abnormality is detected, a means for recording a program according to the taste and preference of a user, a means for interacting with the user by voice, a means for detecting the return of the user by an entrance sensor and reproducing a voice message, and a means for providing an interface for performing remote monitoring and operation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, with the increase in elderly people and single-person households living alone, there is a problem of insufficient safety and comfort in daily life. Furthermore, in the management of homes and buildings, there is a need for efficient and real-time detection of abnormalities. Furthermore, there is a need for appropriate measures to alleviate the sense of loneliness felt by elderly people and people living alone. To solve these issues, it is necessary to introduce a comprehensive monitoring and management system for the living environment. [Means for solving the problem]

[0005] The present invention solves the aforementioned problems by the following means. By providing a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time, it is possible to detect abnormalities and respond quickly. It also has a means for issuing an alarm when an abnormality is detected and automatically calling an ambulance. Furthermore, by recording programs according to the user's preferences and providing a voice dialogue function and a voice message playback function when the user returns home, it aims to make the user's life more comfortable and reduce feelings of loneliness. Additionally, it includes a system that provides an interface for remote monitoring and operation, allowing building managers and family members to understand the situation and respond appropriately.

[0006] "Electricity usage information" is data about the amount and patterns of electricity consumed by homes and buildings.

[0007] "Water usage information" is data about the amount and patterns of water consumed by homes and buildings.

[0008] "Gas usage information" is data about the amount and patterns of gas consumed in a home or building.

[0009] "Webcam footage" means real-time video data captured through a webcam.

[0010] "Real-time collection means" refers to methods or devices that use sensors or cameras to obtain data instantly.

[0011] "Analysis and anomaly detection means" are algorithms or devices that analyze collected data and identify patterns or values ​​that are out of the ordinary.

[0012] A "means for issuing an alert" is a method or device for sending a notification to a user or administrator when an abnormality is detected.

[0013] The "means for recording programs according to the user's tastes and preferences" refers to a method or device for automatically recording television programs based on the user's viewing history and preferences.

[0014] A "means for interacting with a user via voice" is a method or device that uses a generative AI model to recognize speech, generate an appropriate response, and play it back as speech.

[0015] A "doorway sensor" is a sensor installed at the entrance of a house to detect when a user enters or leaves the house.

[0016] A "means for playing back a voice message" is a method or device for recording and storing a voice message and playing it back under certain conditions.

[0017] An "interface for remote monitoring and operation" is a user interface for monitoring and operating the status of a home or building from a remote location via the Internet. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] As an embodiment of the present invention, a system for collecting and analyzing electricity usage information, water usage information, gas usage information and web camera images in real time to detect abnormalities will be specifically described.

[0040] System configuration

[0041] This system consists of the following main components:

[0042] 1. Data Collection Module

[0043] 2. Data Analysis Module

[0044] 3. Anomaly Detection Module

[0045] 4. Warning Module

[0046] 5. User preference recording module

[0047] 6. Voice Dialogue Module

[0048] 7. Remote monitoring interface

[0049] Collection of electricity, water and gas usage information

[0050] The server collects real-time information on electricity, water, and gas usage from smart meters installed in each home or building, allowing the data collection module to centrally manage all types of usage information.

[0051] Webcam footage collection

[0052] The server captures real-time video footage from inside the home via a webcam. This is particularly important for elderly people and those living alone, as video data is crucial for rapid response in emergencies.

[0053] Data analysis and anomaly detection

[0054] The server analyzes collected electricity, water, and gas usage information, as well as webcam footage, in real time. The data analysis module compares past and current data to detect signs of abnormalities with high accuracy. Webcam footage is also analyzed using machine learning technology to detect abnormalities such as the user falling.

[0055] Response after detecting an anomaly

[0056] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[0057] Providing features based on user preferences

[0058] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0059] Voice interaction with the user

[0060] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[0061] Remote Monitoring and Operation

[0062] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[0063] Specific examples

[0064] Example 1: Power anomaly detection and notification

[0065] The server analyzes power usage information in real time, and if abnormally high power usage is detected for more than two consecutive hours, the system immediately notifies the user via voice alerts and push notifications on the mobile app.

[0066] Example 2: Detecting falls among elderly people and notifying emergency services

[0067] If the server analyzes the webcam footage and detects that an elderly person is falling to the floor, the emergency call system will automatically dispatch an ambulance and notify their family members.

[0068] Example 3: Recording programs based on preferences

[0069] The server analyzes the user's viewing history, and if it determines that a new drama series matches the user's taste, it automatically adds the drama to the recording reservation list, so the user does not have to worry about missing it.

[0070] Thus, the invention utilizes advanced technology to provide safety, comfort, and reduced loneliness.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] The server obtains electricity usage information from the smart meter every minute via an API.

[0074] Step 2:

[0075] The server obtains water usage information from the water meter every 10 minutes via an API.

[0076] Step 3:

[0077] The server obtains gas usage information from the gas meter every 15 minutes via an API.

[0078] Step 4:

[0079] The server receives and temporarily stores the video stream from the webcam in real time.

[0080] Step 5:

[0081] The server analyzes the collected power usage information and compares it with data from the past 24 hours to detect abnormally high power usage.

[0082] Step 6:

[0083] The server analyzes water usage information and detects abnormalities when water is used continuously for long periods of time.

[0084] Step 7:

[0085] The server analyzes gas usage information and monitors sudden fluctuations in gas usage in real time.

[0086] Step 8:

[0087] The server analyzes webcam footage using machine learning models to detect abnormal behavior such as the user falling.

[0088] Step 9:

[0089] If the server detects any abnormality, it will immediately send a warning alert to the user.

[0090] Step 10:

[0091] If a fall is detected, the server activates an emergency call system and dispatches an ambulance.

[0092] Step 11:

[0093] The server analyzes the user's viewing history and generates a list of programs to be automatically recorded based on the user's preferences.

[0094] Step 12:

[0095] The server detects that the user has returned home via the entrance sensor and plays a voice message saying "Welcome home."

[0096] Step 13:

[0097] When the user speaks into the microphone, the server converts the voice data into text and generates an appropriate response that is played back aloud.

[0098] Step 14:

[0099] Administrators and family members can access a dedicated remote monitoring interface from the device and check various data in real time.

[0100] Step 15:

[0101] Remotely control electricity, water, and gas usage as needed from your device.

[0102] By performing detailed processing at each step in this way, safety and comfort of the entire system are ensured.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] In modern society, there is a need for systems that allow the elderly and people living alone to quickly respond to abnormalities and emergencies in their daily lives. Furthermore, there is a common need for systems that provide entertainment based on users' hobbies and preferences, streamline daily communication, and enable remote monitoring and operation. Current systems have difficulty providing these functions in an integrated manner, making it essential to provide a comprehensive system that significantly improves safety and convenience.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage using a data analysis module to detect abnormalities, means for issuing an alert when an abnormality is detected, means for recording programs using a user preference analysis module, means for communicating with users via voice using a generative AI model, and means for monitoring and operating through a remote monitoring interface. This enables quick response in emergencies, optimization of entertainment, and improvement of daily communication efficiency while improving the safety and comfort of residents' lives.

[0108] 1. "Electricity usage information" refers to data on the amount and usage patterns of electricity consumed by facilities such as homes and buildings.

[0109] 2. "Water usage information" means data on the amount and patterns of water consumed by households, buildings, and other facilities.

[0110] 3. "Gas usage information" means data relating to the amount and usage patterns of gas consumed in homes, buildings, and other facilities.

[0111] 4. "Webcam footage" means real-time visual data obtained from cameras installed within a home.

[0112] 5. "Data Analysis Module" means an integrated system of software and hardware used to analyze various collected data and detect patterns and anomalies.

[0113] 6. "Anomaly detection means" means a technology that utilizes a data analysis module to recognize data patterns or irregular behavior that exceed certain thresholds in real time.

[0114] 7. "Means of issuing warnings" refers to communication methods such as voice alerts, push notifications, emails, etc., used to notify users and relevant parties when an abnormality is detected.

[0115] 8. "Preference Analysis Module" is a system that analyzes a user's viewing history and behavioral patterns to provide content and services based on the user's preferences.

[0116] 9. “Generative AI model” means an artificial intelligence technology used to generate natural-sounding voice interactions with users and appropriate responses.

[0117] 10. "Remote monitoring interface" means a user interface for monitoring the situation inside a home or facility in real time from an external terminal and performing necessary operations.

[0118] This invention relates to a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities. This system issues warnings based on the results of abnormality detection, records programs based on user preferences, and provides voice interaction. It also includes an interface for remote monitoring and operation.

[0119] System configuration

[0120] The system consists of the following main components:

[0121] 1. Data Collection Module

[0122] 2. Data Analysis Module

[0123] 3. Anomaly Detection Module

[0124] 4. Warning Module

[0125] 5. User preference recording module

[0126] 6. Voice Dialogue Module

[0127] 7. Remote monitoring interface

[0128] Collection of electricity, water and gas usage information

[0129] The server collects real-time information on electricity, water, and gas usage from smart meters installed in each home or building, allowing the data collection module to centrally manage all types of usage information.

[0130] Webcam footage collection

[0131] The server receives real-time video from web cameras in homes, and this video data is especially important for emergency response for elderly users and those living alone.

[0132] Data analysis and anomaly detection

[0133] The server uses a data analysis module to analyze the collected electricity, water, and gas usage information, as well as webcam footage, in real time. This module compares past data with current data to detect signs of abnormalities. Additionally, the webcam footage is analyzed using machine learning technology to detect abnormalities such as the user falling.

[0134] Response after detecting an anomaly

[0135] The server issues a warning if it detects an abnormality. For example, if the water supply has been used continuously for a long period of time, it will notify the user that there may be a water leak. If a fall is detected, the emergency notification system will automatically call an ambulance and notify the user's family.

[0136] Providing features based on user preferences

[0137] The server uses a user preference recording module to analyze the user's past viewing history and preferences, and can automatically record programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0138] Voice interaction with the user

[0139] The server uses a generative AI model to communicate with the user via voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, in response to a question such as "What's the weather like today?", the server can provide weather information.

[0140] Remote Monitoring and Operation

[0141] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when performing regular checks.

[0142] Prompt Sentence Examples

[0143] "Please tell me my name and what the weather is like today."

[0144] "Can you record a new show similar to the one I've been watching lately?"

[0145] "My home's electricity usage seems higher than usual. Is there a problem?"

[0146] In this way, the system of the present invention aims to improve the safety and comfort of residents' lives by using advanced data collection and analysis technology.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1: Data collection

[0149] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building. It also acquires real-time video footage from web cameras installed in homes. The input is data obtained from various smart meters and web cameras, and the output is stored in a database as integrated usage information data and video data.

[0150] Step 2: Initial Data Processing

[0151] The server performs initial processing of the collected data using a data collection module. Specifically, data format conversion and filtering of unnecessary data are performed. The input is the data collected in step 1, and the output is clean data that has been converted into an analyzable format. For example, the units of power usage information can be standardized, and noisy webcam footage can be cleaned up using a noise reduction filter.

[0152] Step 3: Data analysis

[0153] The server analyzes the pre-processed data using a data analysis module. First, an algorithm is applied to detect anomalies by comparing them with past data. Then, a machine learning model is used to detect abnormal behavior from the webcam footage. The input is the pre-processed clean data, and the output is the anomaly detection data as the analysis result. For example, if the power usage information significantly deviates from past usage patterns, it is flagged as an anomaly.

[0154] Step 4: Detect anomalies and send alerts

[0155] If the server detects an anomaly using the anomaly detection module, the warning module is activated. For example, if the water supply is used continuously for a long period of time, a notification is sent to the user to warn that there may be a water leak. If a fall is detected, the emergency notification system is activated to dispatch an ambulance. The input is the anomaly detection data generated in Step 3, and the output is warning information such as a voice alert, a mobile app notification, or an emergency call.

[0156] Step 5: User preference analysis and recording schedule

[0157] The server uses the user preference recording module to analyze the user's past viewing history and preferences. Based on the data collected here, programs that the user may like are automatically added to the recording reservation list. The input is the user's viewing history data, and the output is a new recording reservation list. For example, if a new drama series is determined to match the user's preferences, the drama will be added to the recording reservation list.

[0158] Step 6: Voice interaction

[0159] When a user speaks into the microphone, the voice data is sent to the server. The server uses a generative AI model to analyze the voice data and generate an appropriate response. The input is the voice input data, and the output is the generated voice response. For example, if you say, "Tell me what the weather is today," the generated response will be, "It's sunny today."

[0160] Step 7: Remote monitoring and operation

[0161] Building managers and family members access a dedicated remote monitoring interface from a terminal. The server monitors the situation inside the home in real time and provides operational information. The input is an access request to the remote monitoring interface, and the output is real-time monitoring data and an operational interface. For example, it is possible to display surveillance footage in real time and remotely turn lights and gas on and off.

[0162] (Application example 1)

[0163] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0164] Conventional security systems are limited to monitoring electricity, water, and gas usage information and webcam footage individually, making it difficult to detect abnormalities in real time and respond quickly. Furthermore, they lack functionality based on user preferences and voice interaction, leaving a need for greater convenience. Rapid response in emergencies is especially essential for elderly users and those living alone, and current systems are unable to provide sufficient support.

[0165] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0166] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing a warning when an abnormality is detected, means for recording programs according to the user's preferences, means for communicating with the user via voice, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, and means for detecting activity using a webcam and notifying of an abnormality when activity is detected, thereby improving user safety and comfort.

[0167] "Power usage information" is information for acquiring and analyzing power usage data in real time.

[0168] "Water usage information" refers to information for acquiring and analyzing water usage data in real time.

[0169] "Gas usage information" refers to information for acquiring and analyzing gas usage data in real time.

[0170] "Webcam footage" refers to video data captured in real time through a webcam.

[0171] "Analysis" refers to the analytical processing performed to detect abnormalities based on collected data.

[0172] "Issuing a warning" means notifying the user when an abnormality is detected.

[0173] "Recording a program according to the user's tastes and preferences" means automatically recording a program based on the user's past viewing history and interests.

[0174] "Voice interaction" refers to communication between a user and an interface via voice.

[0175] A "front door sensor" is a device that detects whether the front door is open or closed.

[0176] An "interface for remote monitoring and operation" is an interface that allows the system to be monitored and operated from a remote location via a communication means.

[0177] "Motion detection" is the analysis of webcam footage to detect movement within the footage.

[0178] "Notifying an abnormality" means informing the user, administrator, etc. of a detected abnormality.

[0179] As an embodiment of the present invention, a system for collecting and analyzing electricity usage information, water usage information, gas usage information and web camera images in real time to detect abnormalities will be specifically described.

[0180] System configuration

[0181] This system consists of the following main components:

[0182] 1. Data Collection Module

[0183] 2. Data Analysis Module

[0184] 3. Anomaly Detection Module

[0185] 4. Warning Module

[0186] 5. User preference recording module

[0187] 6. Voice Dialogue Module

[0188] 7. Remote monitoring interface

[0189] 8. Motion Detection Module

[0190] Collection of electricity, water and gas usage information

[0191] The server collects real-time information on electricity, water, and gas usage from smart meters installed in each home or building, allowing the data collection module to centrally manage all types of usage information.

[0192] Webcam footage collection

[0193] The server captures real-time video footage from inside the home via a webcam. This is especially important for elderly people and those living alone, as video data is crucial for rapid response in emergencies.

[0194] Data analysis and anomaly detection

[0195] The server analyzes collected electricity, water, and gas usage information, as well as webcam footage, in real time. The data analysis module compares past and current data to detect signs of abnormalities with high accuracy. Webcam footage is also analyzed using machine learning technology to detect abnormalities such as the user falling.

[0196] Response after detecting an anomaly

[0197] The server issues an alert if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance. The motion detection module analyzes webcam footage and similarly notifies the user if it detects movement.

[0198] Providing features based on user preferences

[0199] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0200] Voice interaction with the user

[0201] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[0202] Remote Monitoring and Operation

[0203] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[0204] Specific examples

[0205] Example 1: Power anomaly detection and notification

[0206] The server analyzes power usage information in real time, and if abnormally high power usage is detected for more than two consecutive hours, the system immediately notifies the user via voice alerts and push notifications on the mobile app.

[0207] Example 2: Detecting falls among elderly people and notifying emergency services

[0208] If the server analyzes the webcam footage and detects that an elderly person is falling to the floor, the emergency call system will automatically dispatch an ambulance and notify their family members.

[0209] Example 3: Recording programs based on preferences

[0210] The server analyzes the user's viewing history, and if it determines that a new drama series matches the user's taste, it automatically adds the drama to the recording reservation list, so the user does not have to worry about missing it.

[0211] Example prompt sentence:

[0212] Please create an app for an anomaly detection system. It will analyze webcam footage in real time and send a notification to a smartphone if an anomaly is detected. Please write pseudocode in Python.

[0213]

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Step 1:

[0216] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building. This information is input into the data collection module and centrally managed. The specific operation of data collection is to periodically obtain the latest usage data from each smart meter and send it to the server. The input is smart meter data, and the output is integrated usage information data.

[0217] Step 2:

[0218] The server collects real-time video from inside the home through a webcam. This video data is stored appropriately by the data collection module and used for analysis. The webcam captures video data at regular intervals and sends it to the server. The input is the video data from the webcam, and the output is the stored video data.

[0219] Step 3:

[0220] The server inputs the collected electricity, water, and gas usage information, as well as webcam footage, into a data analysis module for real-time analysis. The data analysis module compares past and current usage data and applies machine learning algorithms to detect signs of anomalies. For example, it detects an abnormal increase in electricity usage information. The input is past and current usage data and video data, and the output is the analysis results.

[0221] Step 4:

[0222] If the server detects an anomaly based on the analysis results, it issues a warning. The warning module notifies the user of the anomaly through voice alerts, mobile app push notifications, email notifications, etc. Specifically, it generates an appropriate warning message according to the detected anomaly and sends it through the notification means. The input is the analysis result, and the output is the warning message.

[0223] Step 5:

[0224] Based on the user's preferences, the server analyzes the viewing history and automatically schedules recordings of programs that the user may be interested in. The user preference recording module uses a generative AI model based on the viewing history data to predict programs that the user is likely to watch in the future. The input is the user's viewing history data, and the output is a recording schedule list.

[0225] Step 6:

[0226] When a user speaks into the microphone, the server uses a generative AI model to analyze the voice and generate an appropriate response. Specifically, the voice dialogue module converts the voice input into text, and the generative AI model generates an answer based on that text. The input is the user's voice data, and the output is the response voice data.

[0227] Step 7:

[0228] When the door sensor detects the user's return home, the server plays a voice message. The door sensor sends the detection data as input, and the voice interaction module generates and plays an appropriate voice message. The input is the door sensor data, and the output is the voice message to be played.

[0229] Step 8:

[0230] Building managers and family members access a dedicated remote monitoring interface from a terminal, and the server provides an interface that allows them to monitor and control the situation inside the home in real time. This allows for quick response when an abnormality occurs or when regular checks are required. The input is an access request from the monitoring interface, and the output is real-time status information.

[0231] Step 9:

[0232] The server analyzes webcam footage to detect movement. The motion detection module detects specific movements and treats them as an anomaly, issuing a notification if necessary. Specifically, it analyzes changes between video frames to determine whether there is a consistent movement. The input is the webcam video data, and the output is an anomaly notification.

[0233] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0234] As a form for implementing the present invention, we will specifically explain a system that combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities, with an emotion engine that recognizes user emotions.

[0235] System configuration

[0236] This system consists of the following main components:

[0237] 1. Data Collection Module

[0238] 2. Data Analysis Module

[0239] 3. Anomaly Detection Module

[0240] 4. Warning Module

[0241] 5. User preference recording module

[0242] 6. Voice Dialogue Module

[0243] 7. Emotion Engine Module

[0244] 8. Remote monitoring interface

[0245] Collection and analysis of electricity, water and gas usage information

[0246] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building. This allows the data collection module to centrally manage various usage information. The server analyzes the collected data and compares it with past data to detect anomalies.

[0247] Webcam footage collection and analysis

[0248] The server captures real-time video footage from inside the home via a webcam, which is then analyzed using machine learning technology to detect abnormal behavior such as falls.

[0249] Emotion recognition by emotion engine

[0250] The server captures the user's audio and video data and analyzes it with an emotion engine module, which uses machine learning models to analyze the user's facial expressions, tone of voice, and other factors to recognize the user's emotions in real time.

[0251] Response after detecting an anomaly

[0252] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[0253] Emotion-based responses

[0254] The server generates appropriate voice dialogue based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, it can recommend relaxing music. It can also automatically adjust the brightness of lights and music according to the user's emotions.

[0255] Providing features based on user preferences

[0256] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0257] Voice interaction with the user

[0258] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[0259] Remote Monitoring and Operation

[0260] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[0261] Specific examples

[0262] Example 1: Program recommendation using emotion recognition

[0263] If the server recognizes through its emotion engine that the user is feeling stressed, it will recommend relaxing movies and TV shows that suit the user's preferences and automatically record them.

[0264] Example 2: Adjusting the environment based on emotions

[0265] If the server recognizes through its emotion engine that the user is feeling tired, it will dim the lights in the room and play relaxing music.

[0266] Example 3: Emergency response with fall detection and emotion recognition

[0267] If the server detects a fall and the emotion engine recognizes pain or confusion from the user's facial expression, it will immediately activate the emergency call system and dispatch an ambulance.

[0268] In this way, the present invention makes full use of advanced technology to achieve safety, comfort, and optimal response according to the user's emotions.

[0269] The processing flow will be explained below.

[0270] Step 1:

[0271] The server obtains electricity usage information from the smart meter every minute via an API.

[0272] Step 2:

[0273] The server obtains water usage information from the water meter every 10 minutes via an API.

[0274] Step 3:

[0275] The server obtains gas usage information from the gas meter every 15 minutes via an API.

[0276] Step 4:

[0277] The server receives and temporarily stores the video stream from the webcam in real time.

[0278] Step 5:

[0279] The server analyzes the collected power usage information and compares it with data from the past 24 hours to detect abnormally high power usage.

[0280] Step 6:

[0281] The server analyzes water usage information and detects abnormalities when water is used continuously for long periods of time.

[0282] Step 7:

[0283] The server analyzes gas usage information and monitors sudden fluctuations in gas usage in real time.

[0284] Step 8:

[0285] The server analyzes webcam footage using machine learning models to detect abnormal behavior such as the user falling.

[0286] Step 9:

[0287] The server captures the user's voice in real time and analyzes it with an emotion engine to recognize emotions.

[0288] Step 10:

[0289] The server acquires the user's facial expression data from a webcam and analyzes it with an emotion engine to recognize emotions.

[0290] Step 11:

[0291] The server generates appropriate voice dialogue for the user based on the recognized emotion. For example, if the user looks sad, the server might say, "Is there something I can help you with?"

[0292] Step 12:

[0293] Based on the recognized emotions, the server recommends TV programs and movies that match the user's preferences and automatically records them.

[0294] Step 13:

[0295] The server will send a warning alert to the user if an abnormality is detected. For example, if water continues to flow for a long period of time, it will immediately notify the user that there may be a water leak.

[0296] Step 14:

[0297] The server will activate the emergency call system and dispatch an ambulance if a fall is detected, and will also initiate emergency response if the emotion engine detects emotions such as pain or confusion.

[0298] Step 15:

[0299] If the server detects an emotion requiring relaxation through its emotion engine, it will dim the lights slightly and play relaxing music.

[0300] Step 16:

[0301] The server detects that the user has returned home via the entrance sensor and plays a voice message saying "Welcome home."

[0302] Step 17:

[0303] Administrators and family members can access a dedicated remote monitoring interface from the device and check various data in real time.

[0304] Step 18:

[0305] Remotely control electricity, water, and gas usage as needed from your device.

[0306] By performing detailed processing step by step in this way, the safety and comfort of the entire system and optimal responses according to the user's emotions are ensured.

[0307] Example 2

[0308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] Conventional monitoring systems collect electricity, water, and gas usage information and webcam footage, but are limited in their ability to comprehensively analyze this information and detect abnormalities. They also lack the means to analyze a user's emotional state in real time and take appropriate action based on the abnormality or emotion. Furthermore, they lack the functionality to recommend content based on the user's preferences and generate voice dialogue. The present invention aims to solve these problems and provide a safe and comfortable living environment.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0311] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing an alert when an abnormality is detected, means for recording programs according to the user's preferences, means for engaging in voice conversation with the user, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, means for analyzing the user's emotions, means for adjusting the environment based on the user's emotions, and means for generating voice dialogue using a generative AI model. This enables the server to comprehensively analyze electricity, water, and gas usage information and video data, detect abnormalities, and take appropriate measures, as well as adjust the environment, recommend content, and engage in voice dialogue according to the user's emotional state.

[0312] "Electricity usage information" refers to data on the usage of electricity within homes and buildings.

[0313] "Water usage information" refers to data such as the amount of water used within a home or building and the timing of use.

[0314] "Gas usage information" refers to data regarding the amount and timing of gas usage within homes and buildings.

[0315] "Webcam Footage" refers to video footage data collected through a webcam.

[0316] "Means for detecting anomalies" refers to algorithms or analytical systems that identify anomalies by comparing them with typical usage and historical data.

[0317] "Means of issuing an alert" refers to a system or function that sends a notification to a user or administrator when an abnormality is detected.

[0318] "Means for recording programs according to the user's tastes and preferences" refers to a system that automatically records highly relevant programs based on the user's past viewing history and preferences.

[0319] "Means of interacting with the user via voice" refers to a system that uses speech recognition and generative AI models to provide appropriate responses to user utterances.

[0320] "Means for detecting the user's return home using a front door sensor and playing a voice message" refers to a system that detects the user's return home based on information obtained from a front door sensor and plays a pre-set voice message.

[0321] "Means for providing an interface for remote monitoring and operation" refers to an interface that allows building managers and family members to monitor the situation within the home in real time via the Internet and perform necessary operations.

[0322] "Means for analyzing user emotions" refers to a system that analyzes the user's facial expressions and voice data and recognizes their emotional state in real time.

[0323] "Means for adjusting the environment based on the user's emotions" refers to a system that automatically adjusts environmental settings such as room lighting and music according to the user's emotional state.

[0324] "Means for generating a voice dialogue using a generative AI model" refers to a system that uses an AI model to generate a voice dialogue with a user in real time.

[0325] This system combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect anomalies, with an emotion engine that recognizes user emotions. This system consists of the following main components:

[0326] System configuration

[0327] 1. Data Collection Module

[0328] 2. Data Analysis Module

[0329] 3. Anomaly Detection Module

[0330] 4. Warning Module

[0331] 5. User preference recording module

[0332] 6. Voice Dialogue Module

[0333] 7. Emotion Engine Module

[0334] 8. Remote monitoring interface

[0335] Collection and analysis of electricity, water and gas usage information

[0336] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building and manages it centrally. This allows the data collection module to periodically update the usage information. The collected data is stored in a database and processed by the data analysis module. For example, an algorithm is applied to compare the data with past data to detect anomalies in usage.

[0337] Webcam footage collection and analysis

[0338] The server captures real-time video footage from inside the home via a webcam. This video data is analyzed using machine learning technology to detect abnormal behavior, such as falls. If an abnormality is detected, the anomaly detection module immediately issues an alert and, if necessary, activates the emergency call system.

[0339] Emotion recognition by emotion engine

[0340] The server receives the user's audio and video data and analyzes it with an emotion engine module. This emotion engine uses machine learning models to analyze the user's facial expressions, tone of voice, and other factors to recognize their emotional state in real time.

[0341] Response after detecting an anomaly

[0342] The server has a means to issue a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. Also, if a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[0343] Emotion-based responses

[0344] The server generates appropriate voice dialogue based on the user's emotions recognized by the emotion engine. If the user is feeling stressed, it can recommend relaxing music and adjust the lighting in the room. Other features include recommending entertainment content based on the user's emotional state.

[0345] Providing features based on user preferences

[0346] The server analyzes the user's past viewing history and preferences, and has the function of recommending and recording content that the user may be interested in. The recording reservation list is updated in real time to optimize the entertainment experience.

[0347] Voice interaction with the user

[0348] The server uses a generative AI model to converse with the user via voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated in real time. For example, it is possible to have a conversation about everyday information such as the day's schedule, weather, and news.

[0349] Remote Monitoring and Operation

[0350] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to respond quickly and appropriately when an abnormality occurs or regular checks are required.

[0351] Examples and prompts

[0352] Example 1: Program recommendation using emotion recognition

[0353] If the server recognizes through its emotion engine that the user is feeling stressed, it will recommend relaxing movies and TV shows that suit the user's preferences and automatically record them.

[0354] Prompt: "Explain how an emotion recognition engine can recommend relaxing content when it detects stress."

[0355] Example 2: Adjusting the environment based on emotions

[0356] When the server's emotion engine recognizes that the user is feeling tired, it dims the lights in the room and plays relaxing music.

[0357] Prompt: "Describe how you can automatically adjust the environment in a room based on your emotional state."

[0358] Example 3: Emergency response with fall detection and emotion recognition

[0359] If the server detects a fall and the emotion engine recognizes pain or confusion from the user's facial expression, the emergency call system will immediately be activated and an ambulance will be dispatched.

[0360] Prompt: "Please describe your emergency response process when you detect a client has fallen."

[0361] In this way, the present invention utilizes advanced technology to provide optimal responses based on the user's safety, comfort, and emotions.

[0362] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0363] Step 1: Data collection

[0364] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building. This data is input into a data collection module and stored centrally in a database. Specifically, the data sent from the smart meters is obtained using the HTTP protocol and stored as is in the database.

[0365] Input: Real-time data from smart meters

[0366] Output: Electricity, water, and gas usage information stored in a database

[0367] Step 2: Data analysis

[0368] The server analyzes the electricity, water, and gas usage information stored in the database. This involves comparing usage with past data and using algorithms to detect patterns. Specifically, it compares data from the past month and performs statistical analysis to identify sudden increases or decreases in usage.

[0369] Input: Historical electricity, water, and gas usage information stored in a database

[0370] Output: Statistical data for anomaly detection

[0371] Step 3: Anomaly detection

[0372] The server uses the statistical data obtained from the data analysis module to detect anomalies. If an anomaly is detected, the information is input into the anomaly detection module, which generates a warning message. Specifically, if an abnormal pattern is discovered, an alert flag is set and sent to the user notification system.

[0373] Input: Statistical analysis data

[0374] Output: Anomaly detection alert flag and warning message

[0375] Step 4: Collect and analyze webcam footage

[0376] The server collects real-time video data from the webcam and analyzes it using machine learning technology. Specifically, the video analysis algorithm detects abnormal behavior such as falls and issues an alert if necessary.

[0377] Input: Real-time video data from a webcam

[0378] Output: Detected abnormal behavior and warning message

[0379] Step 5: Emotion Recognition

[0380] The server inputs the user's audio and video data into the emotion engine module and analyzes their emotional state. This allows the system to recognize emotions in real time based on the user's facial expressions and tone of voice. Specifically, the emotion engine analyzes the user's audio and video data and generates emotion tags.

[0381] Input: Audio and video data

[0382] Output: Emotion tag

[0383] Step 6: Dealing with abnormalities and emotions

[0384] The server responds appropriately based on the abnormality and emotional state. If an abnormality occurs, the emergency notification system is activated, and depending on the emotional state, the response may include adjusting the music or lighting. Specifically, if an abnormality is detected, a warning message is sent, and if the emotional state is identified, a music playback API is called to play a playlist.

[0385] Input: Anomaly detection alert flag or sentiment tag

[0386] Output: Emergency call, music playback, lighting control

[0387] Step 7: User Preference Management

[0388] The server analyzes the user's past viewing history and preference data, and recommends and records content that the user may be interested in. Specifically, it queries the viewing history database, and the recommendation engine suggests appropriate content.

[0389] Input: Viewing history database

[0390] Output: Recommended content

[0391] Step 8: Voice interaction generation

[0392] The server uses a voice interaction module to communicate with the user via voice. The generative AI model analyzes what the user says and generates an appropriate response. Specifically, the server converts the user's voice input into text, inputs that text into the generative AI model to generate a response, and then converts that response into voice.

[0393] Input: User voice input

[0394] Output: Voice reply

[0395] Step 9: Remote monitoring and operation

[0396] The terminal allows building managers and family members to access the remote monitoring interface and monitor and control the situation inside the home in real time.Specifically, they log in to the remote monitoring interface, check camera footage and smart meter data inside the home, and remotely change lighting and air conditioning settings as needed.

[0397] Input: Real-time monitoring data

[0398] Output: Operation instructions

[0399] (Application example 2)

[0400] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] Conventional anomaly detection systems monitor usage information for electricity, water, gas, etc. in real time, and have the ability to detect abnormalities and issue alarms, but they are unable to grasp the emotional state of the user and take appropriate action.In addition, there are limitations to the means of quickly notifying households and administrators when an abnormality occurs, so further improvements in safety and comfort are needed.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0403] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing a warning when an abnormality is detected, means for recording programs according to a user's preferences, means for audio communication with the user, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, means for recognizing a user's emotion, means for providing warnings and advice based on the user's emotional state, means for detecting abnormalities based on the electricity, water, and gas usage information and sending an electronic message, and means for making an emergency call based on the webcam footage. This enables not only a rapid response when an abnormality is detected but also the provision of advanced warnings and advice based on the user's emotional state.

[0404] - "Electricity usage information" means data indicating the amount of electricity consumed within a home or facility.

[0405] "Water usage information" is data that indicates the amount of water used within a home or facility.

[0406] "Gas usage information" means data indicating the amount of gas used within a home or facility.

[0407] "Webcam footage" refers to real-time video data captured using a webcam.

[0408] "Means for detecting anomalies" refers to analytical means for identifying unusual conditions or behaviors from collected data.

[0409] "Means for issuing warnings" refers to notification means for alerting users and administrators when an abnormality is detected.

[0410] The "means for recording programs according to the user's tastes and preferences" is a means for automatically recording programs and video content that the user likes.

[0411] "Means for communicating with the user by voice" refers to an interface that responds to and communicates with the user by voice.

[0412] The "entrance sensor" is a sensor that detects whether the entrance is open or closed.

[0413] The "means for playing back a voice message" is a means for playing back a voice message when the user returns home.

[0414] "Interface for remote monitoring and operation" refers to an interface that allows a manager or family member in a remote location to monitor and operate the situation within the home in real time.

[0415] "Means for recognizing user emotions" refers to means for identifying the user's emotional state by analyzing the user's facial expressions and voice.

[0416] The "means for providing warnings and advice based on emotional state" refers to means for providing appropriate warnings and advice to a user in accordance with the recognized emotional state.

[0417] "Means for sending electronic messages" refers to means for sending electronic messages to users or administrators when an abnormality is detected.

[0418] "Means for making an emergency call" refers to a means for making an emergency call when an abnormality that poses a threat to human life is detected based on webcam footage.

[0419] As a form for implementing the present invention, we will explain a specific embodiment of a security service that combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities, with an emotion engine that recognizes user emotions.

[0420] System configuration

[0421] This system consists of the following main components:

[0422] 1. Data Collection Module

[0423] The server uses smart meters and web cameras to obtain electricity usage information, water usage information, gas usage information, and real-time images from inside the home.

[0424] 2. Data Analysis Module

[0425] The server analyzes the collected electricity, water, and gas usage information and webcam footage to detect abnormalities, and identifies them by comparing them with past usage data.

[0426] 3. Anomaly Detection Module

[0427] The server will issue a warning to the user if it detects any abnormalities from the analysis results, such as if the water supply has been running for an extended period of time or if it detects abnormal behavior based on webcam footage.

[0428] 4. Emotion Engine Module

[0429] The server recognizes the user's emotions using webcam footage and audio data acquired from a microphone. This emotion engine uses a machine learning model to analyze facial expressions and tone of voice to grasp the user's emotional state in real time.

[0430] 5. Warning Module

[0431] When an abnormality is detected, the server will send a warning to the user via email or push notification. In addition, if the emotion engine recognizes that the user is feeling stressed or anxious, it will provide the user with appropriate advice.

[0432] 6. Voice Dialogue Module

[0433] The server uses a generative AI model to interact with the user through voice: when the user speaks into the microphone, the content is analyzed and an appropriate response is generated.

[0434] 7. Emotion-based responses

[0435] The server can recommend relaxing music or adjust the lighting in the room based on the user's emotions recognized by the emotion engine.

[0436] As a specific example, if a user is feeling stressed, the system may notify them, "You seem to be feeling stressed. I'll play some music to help you relax," and play a relaxing song from the user's music library.

[0437] Examples of prompts to be input to a generative AI model include, "The user's emotions have changed. Please provide a response based on their latest emotional state," and "An anomaly has been detected. Please suggest an appropriate response."

[0438] 8. Remote monitoring interface

[0439] Administrators and family members can access a dedicated remote monitoring interface from a terminal and monitor and operate the system in real time, allowing for immediate response when an abnormality occurs or when regular checks are required.

[0440] This system uses hardware such as smartphones, webcams, and smart meters, and software such as Python, OpenCV, and Keras (with TensorFlow backend). The server collects, analyzes, and processes data from these hardware devices to improve user safety and comfort.

[0441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0442] Step 1:

[0443] Data collection

[0444] The server collects electricity usage information, water usage information, gas usage information, and webcam footage in real time from smart meters and webcams. The input is data from each sensor and device, and the output is real-time data that is stored in a database on the server.

[0445] Step 2:

[0446] Data analysis

[0447] The server compares collected electricity, water, and gas usage information with past data. It also performs face detection and abnormal behavior detection on webcam footage. The input is real-time and past data, and the output is an alert indicating whether or not an abnormality has occurred.

[0448] Step 3:

[0449] Anomaly detection

[0450] The server detects anomalies from the analysis results. If an anomaly is detected, it generates a warning message. The input is the analysis results, and the output is the warning message.

[0451] Step 4:

[0452] emotion recognition

[0453] The server uses the webcam video and microphone audio data to recognize the user's emotions with an emotion engine. The input is real-time video and audio data, and the output is data representing the user's emotional state.

[0454] Step 5:

[0455] Warning

[0456] If an anomaly is detected or if the emotion engine recognizes the user's stress or anxiety, the server issues a warning to the user via email, push notification, etc. The input is the anomaly detection result and emotion recognition result, and the output is email or push notification.

[0457] Step 6:

[0458] Voice dialogue

[0459] The server analyzes what the user says into the microphone using a generative AI model and generates an appropriate response. The input is voice data, and the output is the generated response.

[0460] Step 7:

[0461] Emotion-based responses

[0462] The server recommends music suitable for the user and adjusts lighting based on the recognition results of the emotion engine. The input is emotion recognition data, and the output is commands to play music or control lighting.

[0463] Step 8:

[0464] Remote Monitoring

[0465] Administrators and family members can access a dedicated remote monitoring interface from a terminal to monitor and operate the situation in real time. The input is monitoring data and operation instructions, and the output is the monitoring results and the executed operations.

[0466] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0467] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0468] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0469] [Second embodiment]

[0470] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0471] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0472] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0473] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0474] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0475] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0476] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0477] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0478] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0479] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0480] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0481] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0482] As an embodiment of the present invention, a system for collecting and analyzing electricity usage information, water usage information, gas usage information and web camera images in real time to detect abnormalities will be specifically described.

[0483] System configuration

[0484] This system consists of the following main components:

[0485] 1. Data Collection Module

[0486] 2. Data Analysis Module

[0487] 3. Anomaly Detection Module

[0488] 4. Warning Module

[0489] 5. User preference recording module

[0490] 6. Voice Dialogue Module

[0491] 7. Remote monitoring interface

[0492] Collection of electricity, water and gas usage information

[0493] The server collects real-time information on electricity, water, and gas usage from smart meters installed in each home or building, allowing the data collection module to centrally manage all types of usage information.

[0494] Webcam footage collection

[0495] The server captures real-time video footage from inside the home via a webcam. This is particularly important for elderly people and those living alone, as video data is crucial for rapid response in emergencies.

[0496] Data analysis and anomaly detection

[0497] The server analyzes collected electricity, water, and gas usage information, as well as webcam footage, in real time. The data analysis module compares past and current data to detect signs of abnormalities with high accuracy. Webcam footage is also analyzed using machine learning technology to detect abnormalities such as the user falling.

[0498] Response after detecting an anomaly

[0499] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[0500] Providing features based on user preferences

[0501] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0502] Voice interaction with the user

[0503] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[0504] Remote Monitoring and Operation

[0505] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[0506] Specific examples

[0507] Example 1: Power anomaly detection and notification

[0508] The server analyzes power usage information in real time, and if abnormally high power usage is detected for more than two consecutive hours, the system immediately notifies the user via voice alerts and push notifications on the mobile app.

[0509] Example 2: Detecting falls among elderly people and notifying emergency services

[0510] If the server analyzes the webcam footage and detects that an elderly person is falling to the floor, the emergency call system will automatically dispatch an ambulance and notify their family members.

[0511] Example 3: Recording programs based on preferences

[0512] The server analyzes the user's viewing history, and if it determines that a new drama series matches the user's taste, it automatically adds the drama to the recording reservation list, so the user does not have to worry about missing it.

[0513] Thus, the invention utilizes advanced technology to provide safety, comfort, and reduced loneliness.

[0514] The processing flow will be explained below.

[0515] Step 1:

[0516] The server obtains electricity usage information from the smart meter every minute via an API.

[0517] Step 2:

[0518] The server obtains water usage information from the water meter every 10 minutes via an API.

[0519] Step 3:

[0520] The server obtains gas usage information from the gas meter every 15 minutes via an API.

[0521] Step 4:

[0522] The server receives and temporarily stores the video stream from the webcam in real time.

[0523] Step 5:

[0524] The server analyzes the collected power usage information and compares it with data from the past 24 hours to detect abnormally high power usage.

[0525] Step 6:

[0526] The server analyzes water usage information and detects abnormalities when water is used continuously for long periods of time.

[0527] Step 7:

[0528] The server analyzes gas usage information and monitors sudden fluctuations in gas usage in real time.

[0529] Step 8:

[0530] The server analyzes webcam footage using machine learning models to detect abnormal behavior such as the user falling.

[0531] Step 9:

[0532] If the server detects any abnormality, it will immediately send a warning alert to the user.

[0533] Step 10:

[0534] If a fall is detected, the server activates an emergency call system and dispatches an ambulance.

[0535] Step 11:

[0536] The server analyzes the user's viewing history and generates a list of programs to be automatically recorded based on the user's preferences.

[0537] Step 12:

[0538] The server detects that the user has returned home via the entrance sensor and plays a voice message saying "Welcome home."

[0539] Step 13:

[0540] When the user speaks into the microphone, the server converts the voice data into text and generates an appropriate response that is played back aloud.

[0541] Step 14:

[0542] Administrators and family members can access a dedicated remote monitoring interface from the device and check various data in real time.

[0543] Step 15:

[0544] Remotely control electricity, water, and gas usage as needed from your device.

[0545] By performing detailed processing at each step in this way, safety and comfort of the entire system are ensured.

[0546] Example 1

[0547] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] In modern society, there is a need for systems that allow the elderly and people living alone to quickly respond to abnormalities and emergencies in their daily lives. Furthermore, there is a common need for systems that provide entertainment based on users' hobbies and preferences, streamline daily communication, and enable remote monitoring and operation. Current systems have difficulty providing these functions in an integrated manner, making it essential to provide a comprehensive system that significantly improves safety and convenience.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0550] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage using a data analysis module to detect abnormalities, means for issuing an alert when an abnormality is detected, means for recording programs using a user preference analysis module, means for communicating with users via voice using a generative AI model, and means for monitoring and operating through a remote monitoring interface. This enables quick response in emergencies, optimization of entertainment, and improvement of daily communication efficiency while improving the safety and comfort of residents' lives.

[0551] 1. "Electricity usage information" refers to data on the amount and usage patterns of electricity consumed by facilities such as homes and buildings.

[0552] 2. "Water usage information" means data on the amount and patterns of water consumed by households, buildings, and other facilities.

[0553] 3. "Gas usage information" means data relating to the amount and usage patterns of gas consumed in homes, buildings, and other facilities.

[0554] 4. "Webcam footage" means real-time visual data obtained from cameras installed within a home.

[0555] 5. "Data Analysis Module" means an integrated system of software and hardware used to analyze various collected data and detect patterns and anomalies.

[0556] 6. "Anomaly detection means" means a technology that utilizes a data analysis module to recognize data patterns or irregular behavior that exceed certain thresholds in real time.

[0557] 7. "Means of issuing warnings" refers to communication methods such as voice alerts, push notifications, emails, etc., used to notify users and relevant parties when an abnormality is detected.

[0558] 8. "Preference Analysis Module" is a system that analyzes a user's viewing history and behavioral patterns to provide content and services based on the user's preferences.

[0559] 9. “Generative AI model” means an artificial intelligence technology used to generate natural-sounding voice interactions with users and appropriate responses.

[0560] 10. "Remote monitoring interface" means a user interface for monitoring the situation inside a home or facility in real time from an external terminal and performing necessary operations.

[0561] This invention relates to a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities. This system issues warnings based on the results of abnormality detection, records programs based on user preferences, and provides voice interaction. It also includes an interface for remote monitoring and operation.

[0562] System configuration

[0563] The system consists of the following main components:

[0564] 1. Data Collection Module

[0565] 2. Data Analysis Module

[0566] 3. Anomaly Detection Module

[0567] 4. Warning Module

[0568] 5. User preference recording module

[0569] 6. Voice Dialogue Module

[0570] 7. Remote monitoring interface

[0571] Collection of electricity, water and gas usage information

[0572] The server collects real-time information on electricity, water, and gas usage from smart meters installed in each home or building, allowing the data collection module to centrally manage all types of usage information.

[0573] Webcam footage collection

[0574] The server receives real-time video from web cameras in homes, and this video data is especially important for emergency response for elderly users and those living alone.

[0575] Data analysis and anomaly detection

[0576] The server uses a data analysis module to analyze the collected electricity, water, and gas usage information, as well as webcam footage, in real time. This module compares past data with current data to detect signs of abnormalities. Additionally, the webcam footage is analyzed using machine learning technology to detect abnormalities such as the user falling.

[0577] Response after detecting an anomaly

[0578] The server issues a warning if it detects an abnormality. For example, if the water supply has been used continuously for a long period of time, it will notify the user that there may be a water leak. If a fall is detected, the emergency notification system will automatically call an ambulance and notify the user's family.

[0579] Providing features based on user preferences

[0580] The server uses a user preference recording module to analyze the user's past viewing history and preferences, and can automatically record programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0581] Voice interaction with the user

[0582] The server uses a generative AI model to communicate with the user via voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, in response to a question such as "What's the weather like today?", the server can provide weather information.

[0583] Remote Monitoring and Operation

[0584] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when performing regular checks.

[0585] Prompt Sentence Examples

[0586] "Please tell me my name and what the weather is like today."

[0587] "Can you record a new show similar to the one I've been watching lately?"

[0588] "My home's electricity usage seems higher than usual. Is there a problem?"

[0589] In this way, the system of the present invention aims to improve the safety and comfort of residents' lives by using advanced data collection and analysis technology.

[0590] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0591] Step 1: Data collection

[0592] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building. It also acquires real-time video footage from web cameras installed in homes. The input is data obtained from various smart meters and web cameras, and the output is stored in a database as integrated usage information data and video data.

[0593] Step 2: Initial Data Processing

[0594] The server performs initial processing of the collected data using a data collection module. Specifically, data format conversion and filtering of unnecessary data are performed. The input is the data collected in step 1, and the output is clean data that has been converted into an analyzable format. For example, the units of power usage information can be standardized, and noisy webcam footage can be cleaned up using a noise reduction filter.

[0595] Step 3: Data analysis

[0596] The server analyzes the pre-processed data using a data analysis module. First, an algorithm is applied to detect anomalies by comparing them with past data. Then, a machine learning model is used to detect abnormal behavior from the webcam footage. The input is the pre-processed clean data, and the output is the anomaly detection data as the analysis result. For example, if the power usage information significantly deviates from past usage patterns, it is flagged as an anomaly.

[0597] Step 4: Detect anomalies and send alerts

[0598] If the server detects an anomaly using the anomaly detection module, the warning module is activated. For example, if the water supply is used continuously for a long period of time, a notification is sent to the user to warn that there may be a water leak. If a fall is detected, the emergency notification system is activated to dispatch an ambulance. The input is the anomaly detection data generated in Step 3, and the output is warning information such as a voice alert, a mobile app notification, or an emergency call.

[0599] Step 5: User preference analysis and recording schedule

[0600] The server uses the user preference recording module to analyze the user's past viewing history and preferences. Based on the data collected here, programs that the user may like are automatically added to the recording reservation list. The input is the user's viewing history data, and the output is a new recording reservation list. For example, if a new drama series is determined to match the user's preferences, the drama will be added to the recording reservation list.

[0601] Step 6: Voice interaction

[0602] When a user speaks into the microphone, the voice data is sent to the server. The server uses a generative AI model to analyze the voice data and generate an appropriate response. The input is the voice input data, and the output is the generated voice response. For example, if you say, "Tell me what the weather is today," the generated response will be, "It's sunny today."

[0603] Step 7: Remote monitoring and operation

[0604] Building managers and family members access a dedicated remote monitoring interface from a terminal. The server monitors the situation inside the home in real time and provides operational information. The input is an access request to the remote monitoring interface, and the output is real-time monitoring data and an operational interface. For example, it is possible to display surveillance footage in real time and remotely turn lights and gas on and off.

[0605] (Application example 1)

[0606] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0607] Conventional security systems are limited to monitoring electricity, water, and gas usage information and webcam footage individually, making it difficult to detect abnormalities in real time and respond quickly. Furthermore, they lack functionality based on user preferences and voice interaction, leaving a need for greater convenience. Rapid response in emergencies is especially essential for elderly users and those living alone, and current systems are unable to provide sufficient support.

[0608] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0609] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing a warning when an abnormality is detected, means for recording programs according to the user's preferences, means for communicating with the user via voice, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, and means for detecting activity using a webcam and notifying of an abnormality when activity is detected, thereby improving user safety and comfort.

[0610] "Power usage information" is information for acquiring and analyzing power usage data in real time.

[0611] "Water usage information" refers to information for acquiring and analyzing water usage data in real time.

[0612] "Gas usage information" refers to information for acquiring and analyzing gas usage data in real time.

[0613] "Webcam footage" refers to video data captured in real time through a webcam.

[0614] "Analysis" refers to the analytical processing performed to detect abnormalities based on collected data.

[0615] "Issuing a warning" means notifying the user when an abnormality is detected.

[0616] "Recording a program according to the user's tastes and preferences" means automatically recording a program based on the user's past viewing history and interests.

[0617] "Voice interaction" refers to communication between a user and an interface via voice.

[0618] A "front door sensor" is a device that detects whether the front door is open or closed.

[0619] An "interface for remote monitoring and operation" is an interface that allows the system to be monitored and operated from a remote location via a communication means.

[0620] "Motion detection" is the analysis of webcam footage to detect movement within the footage.

[0621] "Notifying an abnormality" means informing the user, administrator, etc. of a detected abnormality.

[0622] As an embodiment of the present invention, a system for collecting and analyzing electricity usage information, water usage information, gas usage information and web camera images in real time to detect abnormalities will be specifically described.

[0623] System configuration

[0624] This system consists of the following main components:

[0625] 1. Data Collection Module

[0626] 2. Data Analysis Module

[0627] 3. Anomaly Detection Module

[0628] 4. Warning Module

[0629] 5. User preference recording module

[0630] 6. Voice Dialogue Module

[0631] 7. Remote monitoring interface

[0632] 8. Motion Detection Module

[0633] Collection of electricity, water and gas usage information

[0634] The server collects real-time information on electricity, water, and gas usage from smart meters installed in each home or building, allowing the data collection module to centrally manage all types of usage information.

[0635] Webcam footage collection

[0636] The server captures real-time video footage from inside the home via a webcam. This is especially important for elderly people and those living alone, as video data is crucial for rapid response in emergencies.

[0637] Data analysis and anomaly detection

[0638] The server analyzes collected electricity, water, and gas usage information, as well as webcam footage, in real time. The data analysis module compares past and current data to detect signs of abnormalities with high accuracy. Webcam footage is also analyzed using machine learning technology to detect abnormalities such as the user falling.

[0639] Response after detecting an anomaly

[0640] The server issues an alert if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance. The motion detection module analyzes webcam footage and similarly notifies the user if it detects movement.

[0641] Providing features based on user preferences

[0642] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0643] Voice interaction with the user

[0644] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[0645] Remote Monitoring and Operation

[0646] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[0647] Specific examples

[0648] Example 1: Power anomaly detection and notification

[0649] The server analyzes power usage information in real time, and if abnormally high power usage is detected for more than two consecutive hours, the system immediately notifies the user via voice alerts and push notifications on the mobile app.

[0650] Example 2: Detecting falls among elderly people and notifying emergency services

[0651] If the server analyzes the webcam footage and detects that an elderly person is falling to the floor, the emergency call system will automatically dispatch an ambulance and notify their family members.

[0652] Example 3: Recording programs based on preferences

[0653] The server analyzes the user's viewing history, and if it determines that a new drama series matches the user's taste, it automatically adds the drama to the recording reservation list, so the user does not have to worry about missing it.

[0654] Example prompt sentence:

[0655] Please create an app for an anomaly detection system. It will analyze webcam footage in real time and send a notification to a smartphone if an anomaly is detected. Please write pseudocode in Python.

[0656]

[0657] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0658] Step 1:

[0659] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building. This information is input into the data collection module and centrally managed. The specific operation of data collection is to periodically obtain the latest usage data from each smart meter and send it to the server. The input is smart meter data, and the output is integrated usage information data.

[0660] Step 2:

[0661] The server collects real-time video from inside the home through a webcam. This video data is stored appropriately by the data collection module and used for analysis. The webcam captures video data at regular intervals and sends it to the server. The input is the video data from the webcam, and the output is the stored video data.

[0662] Step 3:

[0663] The server inputs the collected electricity, water, and gas usage information, as well as webcam footage, into a data analysis module for real-time analysis. The data analysis module compares past and current usage data and applies machine learning algorithms to detect signs of anomalies. For example, it detects an abnormal increase in electricity usage information. The input is past and current usage data and video data, and the output is the analysis results.

[0664] Step 4:

[0665] If the server detects an anomaly based on the analysis results, it issues a warning. The warning module notifies the user of the anomaly through voice alerts, mobile app push notifications, email notifications, etc. Specifically, it generates an appropriate warning message according to the detected anomaly and sends it through the notification means. The input is the analysis result, and the output is the warning message.

[0666] Step 5:

[0667] Based on the user's preferences, the server analyzes the viewing history and automatically schedules recordings of programs that the user may be interested in. The user preference recording module uses a generative AI model based on the viewing history data to predict programs that the user is likely to watch in the future. The input is the user's viewing history data, and the output is a recording schedule list.

[0668] Step 6:

[0669] When a user speaks into the microphone, the server uses a generative AI model to analyze the voice and generate an appropriate response. Specifically, the voice dialogue module converts the voice input into text, and the generative AI model generates an answer based on that text. The input is the user's voice data, and the output is the response voice data.

[0670] Step 7:

[0671] When the door sensor detects the user's return home, the server plays a voice message. The door sensor sends the detection data as input, and the voice interaction module generates and plays an appropriate voice message. The input is the door sensor data, and the output is the voice message to be played.

[0672] Step 8:

[0673] Building managers and family members access a dedicated remote monitoring interface from a terminal, and the server provides an interface that allows them to monitor and control the situation inside the home in real time. This allows for quick response when an abnormality occurs or when regular checks are required. The input is an access request from the monitoring interface, and the output is real-time status information.

[0674] Step 9:

[0675] The server analyzes webcam footage to detect movement. The motion detection module detects specific movements and treats them as an anomaly, issuing a notification if necessary. Specifically, it analyzes changes between video frames to determine whether there is a consistent movement. The input is the webcam video data, and the output is an anomaly notification.

[0676] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0677] As a form for implementing the present invention, we will specifically explain a system that combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities, with an emotion engine that recognizes user emotions.

[0678] System configuration

[0679] This system consists of the following main components:

[0680] 1. Data Collection Module

[0681] 2. Data Analysis Module

[0682] 3. Anomaly Detection Module

[0683] 4. Warning Module

[0684] 5. User preference recording module

[0685] 6. Voice Dialogue Module

[0686] 7. Emotion Engine Module

[0687] 8. Remote monitoring interface

[0688] Collection and analysis of electricity, water and gas usage information

[0689] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building. This allows the data collection module to centrally manage various usage information. The server analyzes the collected data and compares it with past data to detect anomalies.

[0690] Webcam footage collection and analysis

[0691] The server captures real-time video footage from inside the home via a webcam, which is then analyzed using machine learning technology to detect abnormal behavior such as falls.

[0692] Emotion recognition by emotion engine

[0693] The server captures the user's audio and video data and analyzes it with an emotion engine module, which uses machine learning models to analyze the user's facial expressions, tone of voice, and other factors to recognize the user's emotions in real time.

[0694] Response after detecting an anomaly

[0695] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[0696] Emotion-based responses

[0697] The server generates appropriate voice dialogue based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, it can recommend relaxing music. It can also automatically adjust the brightness of lights and music according to the user's emotions.

[0698] Providing features based on user preferences

[0699] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0700] Voice interaction with the user

[0701] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[0702] Remote Monitoring and Operation

[0703] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[0704] Specific examples

[0705] Example 1: Program recommendation using emotion recognition

[0706] If the server recognizes through its emotion engine that the user is feeling stressed, it will recommend relaxing movies and TV shows that suit the user's preferences and automatically record them.

[0707] Example 2: Adjusting the environment based on emotions

[0708] If the server recognizes through its emotion engine that the user is feeling tired, it will dim the lights in the room and play relaxing music.

[0709] Example 3: Emergency response with fall detection and emotion recognition

[0710] If the server detects a fall and the emotion engine recognizes pain or confusion from the user's facial expression, it will immediately activate the emergency call system and dispatch an ambulance.

[0711] In this way, the present invention makes full use of advanced technology to achieve safety, comfort, and optimal response according to the user's emotions.

[0712] The processing flow will be explained below.

[0713] Step 1:

[0714] The server obtains electricity usage information from the smart meter every minute via an API.

[0715] Step 2:

[0716] The server obtains water usage information from the water meter every 10 minutes via an API.

[0717] Step 3:

[0718] The server obtains gas usage information from the gas meter every 15 minutes via an API.

[0719] Step 4:

[0720] The server receives and temporarily stores the video stream from the webcam in real time.

[0721] Step 5:

[0722] The server analyzes the collected power usage information and compares it with data from the past 24 hours to detect abnormally high power usage.

[0723] Step 6:

[0724] The server analyzes water usage information and detects abnormalities when water is used continuously for long periods of time.

[0725] Step 7:

[0726] The server analyzes gas usage information and monitors sudden fluctuations in gas usage in real time.

[0727] Step 8:

[0728] The server analyzes webcam footage using machine learning models to detect abnormal behavior such as the user falling.

[0729] Step 9:

[0730] The server captures the user's voice in real time and analyzes it with an emotion engine to recognize emotions.

[0731] Step 10:

[0732] The server acquires the user's facial expression data from a webcam and analyzes it with an emotion engine to recognize emotions.

[0733] Step 11:

[0734] The server generates appropriate voice dialogue for the user based on the recognized emotion. For example, if the user looks sad, the server might say, "Is there something I can help you with?"

[0735] Step 12:

[0736] Based on the recognized emotions, the server recommends TV programs and movies that match the user's preferences and automatically records them.

[0737] Step 13:

[0738] The server will send a warning alert to the user if an abnormality is detected. For example, if water continues to flow for a long period of time, it will immediately notify the user that there may be a water leak.

[0739] Step 14:

[0740] The server will activate the emergency call system and dispatch an ambulance if a fall is detected, and will also initiate emergency response if the emotion engine detects emotions such as pain or confusion.

[0741] Step 15:

[0742] If the server detects an emotion requiring relaxation through its emotion engine, it will dim the lights slightly and play relaxing music.

[0743] Step 16:

[0744] The server detects that the user has returned home via the entrance sensor and plays a voice message saying "Welcome home."

[0745] Step 17:

[0746] Administrators and family members can access a dedicated remote monitoring interface from the device and check various data in real time.

[0747] Step 18:

[0748] Remotely control electricity, water, and gas usage as needed from your device.

[0749] By performing detailed processing step by step in this way, the safety and comfort of the entire system and optimal responses according to the user's emotions are ensured.

[0750] Example 2

[0751] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0752] Conventional monitoring systems collect electricity, water, and gas usage information and webcam footage, but are limited in their ability to comprehensively analyze this information and detect abnormalities. They also lack the means to analyze a user's emotional state in real time and take appropriate action based on the abnormality or emotion. Furthermore, they lack the functionality to recommend content based on the user's preferences and generate voice dialogue. The present invention aims to solve these problems and provide a safe and comfortable living environment.

[0753] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0754] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing an alert when an abnormality is detected, means for recording programs according to the user's preferences, means for engaging in voice conversation with the user, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, means for analyzing the user's emotions, means for adjusting the environment based on the user's emotions, and means for generating voice dialogue using a generative AI model. This enables the server to comprehensively analyze electricity, water, and gas usage information and video data, detect abnormalities, and take appropriate measures, as well as adjust the environment, recommend content, and engage in voice dialogue according to the user's emotional state.

[0755] "Electricity usage information" refers to data on the usage of electricity within homes and buildings.

[0756] "Water usage information" refers to data such as the amount of water used within a home or building and the timing of use.

[0757] "Gas usage information" refers to data regarding the amount and timing of gas usage within homes and buildings.

[0758] "Webcam Footage" refers to video footage data collected through a webcam.

[0759] "Means for detecting anomalies" refers to algorithms or analytical systems that identify anomalies by comparing them with typical usage and historical data.

[0760] "Means of issuing an alert" refers to a system or function that sends a notification to a user or administrator when an abnormality is detected.

[0761] "Means for recording programs according to the user's tastes and preferences" refers to a system that automatically records highly relevant programs based on the user's past viewing history and preferences.

[0762] "Means of interacting with the user via voice" refers to a system that uses speech recognition and generative AI models to provide appropriate responses to user utterances.

[0763] "Means for detecting the user's return home using a front door sensor and playing a voice message" refers to a system that detects the user's return home based on information obtained from a front door sensor and plays a pre-set voice message.

[0764] "Means for providing an interface for remote monitoring and operation" refers to an interface that allows building managers and family members to monitor the situation within the home in real time via the Internet and perform necessary operations.

[0765] "Means for analyzing user emotions" refers to a system that analyzes the user's facial expressions and voice data and recognizes their emotional state in real time.

[0766] "Means for adjusting the environment based on the user's emotions" refers to a system that automatically adjusts environmental settings such as room lighting and music according to the user's emotional state.

[0767] "Means for generating a voice dialogue using a generative AI model" refers to a system that uses an AI model to generate a voice dialogue with a user in real time.

[0768] This system combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect anomalies, with an emotion engine that recognizes user emotions. This system consists of the following main components:

[0769] System configuration

[0770] 1. Data Collection Module

[0771] 2. Data Analysis Module

[0772] 3. Anomaly Detection Module

[0773] 4. Warning Module

[0774] 5. User preference recording module

[0775] 6. Voice Dialogue Module

[0776] 7. Emotion Engine Module

[0777] 8. Remote monitoring interface

[0778] Collection and analysis of electricity, water and gas usage information

[0779] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building and manages it centrally. This allows the data collection module to periodically update the usage information. The collected data is stored in a database and processed by the data analysis module. For example, an algorithm is applied to compare the data with past data to detect anomalies in usage.

[0780] Webcam footage collection and analysis

[0781] The server captures real-time video footage from inside the home via a webcam. This video data is analyzed using machine learning technology to detect abnormal behavior, such as falls. If an abnormality is detected, the anomaly detection module immediately issues an alert and, if necessary, activates the emergency call system.

[0782] Emotion recognition by emotion engine

[0783] The server receives the user's audio and video data and analyzes it with an emotion engine module. This emotion engine uses machine learning models to analyze the user's facial expressions, tone of voice, and other factors to recognize their emotional state in real time.

[0784] Response after detecting an anomaly

[0785] The server has a means to issue a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. Also, if a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[0786] Emotion-based responses

[0787] The server generates appropriate voice dialogue based on the user's emotions recognized by the emotion engine. If the user is feeling stressed, it can recommend relaxing music and adjust the lighting in the room. Other features include recommending entertainment content based on the user's emotional state.

[0788] Providing features based on user preferences

[0789] The server analyzes the user's past viewing history and preferences, and has the function of recommending and recording content that the user may be interested in. The recording reservation list is updated in real time to optimize the entertainment experience.

[0790] Voice interaction with the user

[0791] The server uses a generative AI model to converse with the user via voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated in real time. For example, it is possible to have a conversation about everyday information such as the day's schedule, weather, and news.

[0792] Remote Monitoring and Operation

[0793] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to respond quickly and appropriately when an abnormality occurs or regular checks are required.

[0794] Examples and prompts

[0795] Example 1: Program recommendation using emotion recognition

[0796] If the server recognizes through its emotion engine that the user is feeling stressed, it will recommend relaxing movies and TV shows that suit the user's preferences and automatically record them.

[0797] Prompt: "Explain how an emotion recognition engine can recommend relaxing content when it detects stress."

[0798] Example 2: Adjusting the environment based on emotions

[0799] When the server's emotion engine recognizes that the user is feeling tired, it dims the lights in the room and plays relaxing music.

[0800] Prompt: "Describe how you can automatically adjust the environment in a room based on your emotional state."

[0801] Example 3: Emergency response with fall detection and emotion recognition

[0802] If the server detects a fall and the emotion engine recognizes pain or confusion from the user's facial expression, the emergency call system will immediately be activated and an ambulance will be dispatched.

[0803] Prompt: "Please describe your emergency response process when you detect a client has fallen."

[0804] In this way, the present invention utilizes advanced technology to provide optimal responses based on the user's safety, comfort, and emotions.

[0805] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0806] Step 1: Data collection

[0807] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building. This data is input into a data collection module and stored centrally in a database. Specifically, the data sent from the smart meters is obtained using the HTTP protocol and stored as is in the database.

[0808] Input: Real-time data from smart meters

[0809] Output: Electricity, water, and gas usage information stored in a database

[0810] Step 2: Data analysis

[0811] The server analyzes the electricity, water, and gas usage information stored in the database. This involves comparing usage with past data and using algorithms to detect patterns. Specifically, it compares data from the past month and performs statistical analysis to identify sudden increases or decreases in usage.

[0812] Input: Historical electricity, water, and gas usage information stored in a database

[0813] Output: Statistical data for anomaly detection

[0814] Step 3: Anomaly detection

[0815] The server uses the statistical data obtained from the data analysis module to detect anomalies. If an anomaly is detected, the information is input into the anomaly detection module, which generates a warning message. Specifically, if an abnormal pattern is discovered, an alert flag is set and sent to the user notification system.

[0816] Input: Statistical analysis data

[0817] Output: Anomaly detection alert flag and warning message

[0818] Step 4: Collect and analyze webcam footage

[0819] The server collects real-time video data from the webcam and analyzes it using machine learning technology. Specifically, the video analysis algorithm detects abnormal behavior such as falls and issues an alert if necessary.

[0820] Input: Real-time video data from a webcam

[0821] Output: Detected abnormal behavior and warning message

[0822] Step 5: Emotion Recognition

[0823] The server inputs the user's audio and video data into the emotion engine module and analyzes their emotional state. This allows the system to recognize emotions in real time based on the user's facial expressions and tone of voice. Specifically, the emotion engine analyzes the user's audio and video data and generates emotion tags.

[0824] Input: Audio and video data

[0825] Output: Emotion tag

[0826] Step 6: Dealing with abnormalities and emotions

[0827] The server responds appropriately based on the abnormality and emotional state. If an abnormality occurs, the emergency notification system is activated, and depending on the emotional state, the response may include adjusting the music or lighting. Specifically, if an abnormality is detected, a warning message is sent, and if the emotional state is identified, a music playback API is called to play a playlist.

[0828] Input: Anomaly detection alert flag or sentiment tag

[0829] Output: Emergency call, music playback, lighting control

[0830] Step 7: User Preference Management

[0831] The server analyzes the user's past viewing history and preference data, and recommends and records content that the user may be interested in. Specifically, it queries the viewing history database, and the recommendation engine suggests appropriate content.

[0832] Input: Viewing history database

[0833] Output: Recommended content

[0834] Step 8: Voice interaction generation

[0835] The server uses a voice interaction module to communicate with the user via voice. The generative AI model analyzes what the user says and generates an appropriate response. Specifically, the server converts the user's voice input into text, inputs that text into the generative AI model to generate a response, and then converts that response into voice.

[0836] Input: User voice input

[0837] Output: Voice reply

[0838] Step 9: Remote monitoring and operation

[0839] The terminal allows building managers and family members to access the remote monitoring interface and monitor and control the situation inside the home in real time.Specifically, they log in to the remote monitoring interface, check camera footage and smart meter data inside the home, and remotely change lighting and air conditioning settings as needed.

[0840] Input: Real-time monitoring data

[0841] Output: Operation instructions

[0842] (Application example 2)

[0843] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0844] Conventional anomaly detection systems monitor usage information for electricity, water, gas, etc. in real time, and have the ability to detect abnormalities and issue alarms, but they are unable to grasp the emotional state of the user and take appropriate action.In addition, there are limitations to the means of quickly notifying households and administrators when an abnormality occurs, so further improvements in safety and comfort are needed.

[0845] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0846] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing a warning when an abnormality is detected, means for recording programs according to a user's preferences, means for audio communication with the user, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, means for recognizing a user's emotion, means for providing warnings and advice based on the user's emotional state, means for detecting abnormalities based on the electricity, water, and gas usage information and sending an electronic message, and means for making an emergency call based on the webcam footage. This enables not only a rapid response when an abnormality is detected but also the provision of advanced warnings and advice based on the user's emotional state.

[0847] - "Electricity usage information" means data indicating the amount of electricity consumed within a home or facility.

[0848] "Water usage information" is data that indicates the amount of water used within a home or facility.

[0849] "Gas usage information" means data indicating the amount of gas used within a home or facility.

[0850] "Webcam footage" refers to real-time video data captured using a webcam.

[0851] "Means for detecting anomalies" refers to analytical means for identifying unusual conditions or behaviors from collected data.

[0852] "Means for issuing warnings" refers to notification means for alerting users and administrators when an abnormality is detected.

[0853] The "means for recording programs according to the user's tastes and preferences" is a means for automatically recording programs and video content that the user likes.

[0854] "Means for communicating with the user by voice" refers to an interface that responds to and communicates with the user by voice.

[0855] The "entrance sensor" is a sensor that detects whether the entrance is open or closed.

[0856] The "means for playing back a voice message" is a means for playing back a voice message when the user returns home.

[0857] "Interface for remote monitoring and operation" refers to an interface that allows a manager or family member in a remote location to monitor and operate the situation within the home in real time.

[0858] "Means for recognizing user emotions" refers to means for identifying the user's emotional state by analyzing the user's facial expressions and voice.

[0859] The "means for providing warnings and advice based on emotional state" refers to means for providing appropriate warnings and advice to a user in accordance with the recognized emotional state.

[0860] "Means for sending electronic messages" refers to means for sending electronic messages to users or administrators when an abnormality is detected.

[0861] "Means for making an emergency call" refers to a means for making an emergency call when an abnormality that poses a threat to human life is detected based on webcam footage.

[0862] As a form for implementing the present invention, we will explain a specific embodiment of a security service that combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities, with an emotion engine that recognizes user emotions.

[0863] System configuration

[0864] This system consists of the following main components:

[0865] 1. Data Collection Module

[0866] The server uses smart meters and web cameras to obtain electricity usage information, water usage information, gas usage information, and real-time images from inside the home.

[0867] 2. Data Analysis Module

[0868] The server analyzes the collected electricity, water, and gas usage information and webcam footage to detect abnormalities, and identifies them by comparing them with past usage data.

[0869] 3. Anomaly Detection Module

[0870] The server will issue a warning to the user if it detects any abnormalities from the analysis results, such as if the water supply has been running for an extended period of time or if it detects abnormal behavior based on webcam footage.

[0871] 4. Emotion Engine Module

[0872] The server recognizes the user's emotions using webcam footage and audio data acquired from a microphone. This emotion engine uses a machine learning model to analyze facial expressions and tone of voice to grasp the user's emotional state in real time.

[0873] 5. Warning Module

[0874] When an abnormality is detected, the server will send a warning to the user via email or push notification. In addition, if the emotion engine recognizes that the user is feeling stressed or anxious, it will provide the user with appropriate advice.

[0875] 6. Voice Dialogue Module

[0876] The server uses a generative AI model to interact with the user through voice: when the user speaks into the microphone, the content is analyzed and an appropriate response is generated.

[0877] 7. Emotion-based responses

[0878] The server can recommend relaxing music or adjust the lighting in the room based on the user's emotions recognized by the emotion engine.

[0879] As a specific example, if a user is feeling stressed, the system may notify them, "You seem to be feeling stressed. I'll play some music to help you relax," and play a relaxing song from the user's music library.

[0880] Examples of prompts to be input to a generative AI model include, "The user's emotions have changed. Please provide a response based on their latest emotional state," and "An anomaly has been detected. Please suggest an appropriate response."

[0881] 8. Remote monitoring interface

[0882] Administrators and family members can access a dedicated remote monitoring interface from a terminal and monitor and operate the system in real time, allowing for immediate response when an abnormality occurs or when regular checks are required.

[0883] This system uses hardware such as smartphones, webcams, and smart meters, and software such as Python, OpenCV, and Keras (with TensorFlow backend). The server collects, analyzes, and processes data from these hardware devices to improve user safety and comfort.

[0884] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0885] Step 1:

[0886] Data collection

[0887] The server collects electricity usage information, water usage information, gas usage information, and webcam footage in real time from smart meters and webcams. The input is data from each sensor and device, and the output is real-time data that is stored in a database on the server.

[0888] Step 2:

[0889] Data analysis

[0890] The server compares collected electricity, water, and gas usage information with past data. It also performs face detection and abnormal behavior detection on webcam footage. The input is real-time and past data, and the output is an alert indicating whether or not an abnormality has occurred.

[0891] Step 3:

[0892] Anomaly detection

[0893] The server detects anomalies from the analysis results. If an anomaly is detected, it generates a warning message. The input is the analysis results, and the output is the warning message.

[0894] Step 4:

[0895] emotion recognition

[0896] The server uses the webcam video and microphone audio data to recognize the user's emotions with an emotion engine. The input is real-time video and audio data, and the output is data representing the user's emotional state.

[0897] Step 5:

[0898] Warning

[0899] If an anomaly is detected or if the emotion engine recognizes the user's stress or anxiety, the server issues a warning to the user via email, push notification, etc. The input is the anomaly detection result and emotion recognition result, and the output is email or push notification.

[0900] Step 6:

[0901] Voice dialogue

[0902] The server analyzes what the user says into the microphone using a generative AI model and generates an appropriate response. The input is voice data, and the output is the generated response.

[0903] Step 7:

[0904] Emotion-based responses

[0905] The server recommends music suitable for the user and adjusts lighting based on the recognition results of the emotion engine. The input is emotion recognition data, and the output is commands to play music or control lighting.

[0906] Step 8:

[0907] Remote Monitoring

[0908] Administrators and family members can access a dedicated remote monitoring interface from a terminal to monitor and operate the situation in real time. The input is monitoring data and operation instructions, and the output is the monitoring results and the executed operations.

[0909] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0910] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0911] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0912] [Third embodiment]

[0913] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0914] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0915] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0916] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0917] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0918] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0919] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0920] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0921] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0922] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0923] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0924] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0925] As an embodiment of the present invention, a system for collecting and analyzing electricity usage information, water usage information, gas usage information and web camera images in real time to detect abnormalities will be specifically described.

[0926] System configuration

[0927] This system consists of the following main components:

[0928] 1. Data Collection Module

[0929] 2. Data Analysis Module

[0930] 3. Anomaly Detection Module

[0931] 4. Warning Module

[0932] 5. User preference recording module

[0933] 6. Voice Dialogue Module

[0934] 7. Remote monitoring interface

[0935] Collection of electricity, water and gas usage information

[0936] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building, allowing the data collection module to centrally manage various types of usage information.

[0937] Webcam footage collection

[0938] The server captures real-time video footage from inside the home via a webcam. This is particularly important for elderly people and those living alone, as video data is crucial for rapid response in emergencies.

[0939] Data analysis and anomaly detection

[0940] The server analyzes collected electricity, water, and gas usage information, as well as webcam footage, in real time. The data analysis module compares past and current data to detect signs of abnormalities with high accuracy. Webcam footage is also analyzed using machine learning technology to detect abnormalities such as the user falling.

[0941] Response after detecting an anomaly

[0942] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[0943] Providing features based on user preferences

[0944] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[0945] Voice interaction with the user

[0946] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[0947] Remote Monitoring and Operation

[0948] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[0949] Specific examples

[0950] Example 1: Power anomaly detection and notification

[0951] The server analyzes power usage information in real time, and if abnormally high power usage is detected for more than two consecutive hours, the system immediately notifies the user via voice alerts and push notifications on the mobile app.

[0952] Example 2: Detecting falls among elderly people and calling an emergency services

[0953] If the server analyzes the webcam footage and detects that an elderly person is falling to the floor, the emergency call system will automatically dispatch an ambulance and notify their family members.

[0954] Example 3: Recording programs based on preferences

[0955] The server analyzes the user's viewing history, and if it determines that a new drama series matches the user's taste, it automatically adds the drama to the recording reservation list, so the user does not have to worry about missing it.

[0956] Thus, the invention utilizes advanced technology to provide safety, comfort, and reduced loneliness.

[0957] The processing flow will be explained below.

[0958] Step 1:

[0959] The server obtains electricity usage information from the smart meter every minute via an API.

[0960] Step 2:

[0961] The server obtains water usage information from the water meter every 10 minutes via an API.

[0962] Step 3:

[0963] The server obtains gas usage information from the gas meter every 15 minutes via an API.

[0964] Step 4:

[0965] The server receives and temporarily stores the video stream from the webcam in real time.

[0966] Step 5:

[0967] The server analyzes the collected power usage information and compares it with data from the past 24 hours to detect abnormally high power usage.

[0968] Step 6:

[0969] The server analyzes water usage information and detects abnormalities when water is used continuously for long periods of time.

[0970] Step 7:

[0971] The server analyzes gas usage information and monitors sudden fluctuations in gas usage in real time.

[0972] Step 8:

[0973] The server analyzes webcam footage using machine learning models to detect abnormal behavior such as the user falling.

[0974] Step 9:

[0975] If the server detects any abnormality, it will immediately send a warning alert to the user.

[0976] Step 10:

[0977] If a fall is detected, the server activates an emergency call system and dispatches an ambulance.

[0978] Step 11:

[0979] The server analyzes the user's viewing history and generates a list of programs to be automatically recorded based on the user's preferences.

[0980] Step 12:

[0981] The server detects that the user has returned home via the entrance sensor and plays a voice message saying "Welcome home."

[0982] Step 13:

[0983] When the user speaks into the microphone, the server converts the voice data into text and generates an appropriate response that is played back aloud.

[0984] Step 14:

[0985] Administrators and family members can access a dedicated remote monitoring interface from the device and check various data in real time.

[0986] Step 15:

[0987] Remotely control electricity, water, and gas usage as needed from your device.

[0988] By performing detailed processing at each step in this way, safety and comfort of the entire system are ensured.

[0989] Example 1

[0990] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0991] In modern society, there is a need for systems that allow the elderly and people living alone to quickly respond to abnormalities and emergencies in their daily lives. Furthermore, there is a common need for systems that provide entertainment based on users' hobbies and preferences, streamline daily communication, and enable remote monitoring and operation. Current systems have difficulty providing these functions in an integrated manner, making it essential to provide a comprehensive system that significantly improves safety and convenience.

[0992] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0993] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage using a data analysis module to detect abnormalities, means for issuing an alert when an abnormality is detected, means for recording programs using a user preference analysis module, means for communicating with users via voice using a generative AI model, and means for monitoring and operating through a remote monitoring interface. This enables quick response in emergencies, optimization of entertainment, and improvement of daily communication efficiency while improving the safety and comfort of residents' lives.

[0994] 1. "Electricity usage information" refers to data on the amount and usage patterns of electricity consumed by facilities such as homes and buildings.

[0995] 2. "Water usage information" means data on the amount and patterns of water consumed by households, buildings, and other facilities.

[0996] 3. "Gas usage information" means data relating to the amount and usage patterns of gas consumed in homes, buildings, and other facilities.

[0997] 4. "Webcam footage" means real-time visual data obtained from cameras installed within a home.

[0998] 5. "Data Analysis Module" means an integrated system of software and hardware used to analyze various collected data and detect patterns and anomalies.

[0999] 6. "Anomaly detection means" means a technology that utilizes a data analysis module to recognize data patterns or irregular behavior that exceed certain thresholds in real time.

[1000] 7. "Means of issuing warnings" refers to communication methods such as voice alerts, push notifications, emails, etc., used to notify users and relevant parties when an abnormality is detected.

[1001] 8. "Preference Analysis Module" is a system that analyzes a user's viewing history and behavioral patterns and provides content and services based on the user's preferences.

[1002] 9. “Generative AI model” means an artificial intelligence technology used to generate natural-sounding voice interactions with users and appropriate responses.

[1003] 10. "Remote monitoring interface" means a user interface for monitoring the situation inside a home or facility in real time from an external terminal and performing necessary operations.

[1004] This invention relates to a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities. This system issues warnings based on the results of abnormality detection, records programs based on user preferences, and provides voice interaction. It also includes an interface for remote monitoring and operation.

[1005] System configuration

[1006] The system consists of the following main components:

[1007] 1. Data Collection Module

[1008] 2. Data Analysis Module

[1009] 3. Anomaly Detection Module

[1010] 4. Warning Module

[1011] 5. User preference recording module

[1012] 6. Voice Dialogue Module

[1013] 7. Remote monitoring interface

[1014] Collection of electricity, water and gas usage information

[1015] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building, allowing the data collection module to centrally manage various types of usage information.

[1016] Webcam footage collection

[1017] The server receives real-time video from web cameras in homes, and this video data is especially important for emergency response for elderly users and those living alone.

[1018] Data analysis and anomaly detection

[1019] The server uses a data analysis module to analyze the collected electricity, water, and gas usage information, as well as webcam footage, in real time. This module compares past data with current data to detect signs of abnormalities. Additionally, the webcam footage is analyzed using machine learning technology to detect abnormalities such as the user falling.

[1020] Response after detecting an anomaly

[1021] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it will notify the user that there may be a water leak. If a fall is detected, the emergency notification system will automatically call an ambulance and notify the user's family.

[1022] Providing features based on user preferences

[1023] The server uses a user preference recording module to analyze the user's past viewing history and preferences, and can automatically record programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[1024] Voice interaction with the user

[1025] The server uses a generative AI model to communicate with the user via voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, in response to a question such as "What's the weather like today?", the server can provide weather information.

[1026] Remote Monitoring and Operation

[1027] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when performing regular checks.

[1028] Prompt Sentence Examples

[1029] "Please tell me my name and what the weather is like today."

[1030] "Can you record a new show similar to the one I've been watching lately?"

[1031] "My home's electricity usage seems higher than usual. Is there a problem?"

[1032] In this way, the system of the present invention aims to improve the safety and comfort of residents' lives by using advanced data collection and analysis technology.

[1033] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1034] Step 1: Data collection

[1035] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building. It also acquires real-time video footage from web cameras installed in homes. The input is data obtained from various smart meters and web cameras, and the output is stored in a database as integrated usage information data and video data.

[1036] Step 2: Initial Data Processing

[1037] The server performs initial processing of the collected data using a data collection module. Specifically, data format conversion and filtering of unnecessary data are performed. The input is the data collected in step 1, and the output is clean data that has been converted into an analyzable format. For example, the units of power usage information can be standardized, and noisy webcam footage can be cleaned up using a noise reduction filter.

[1038] Step 3: Data analysis

[1039] The server analyzes the pre-processed data using a data analysis module. First, an algorithm is applied to detect anomalies by comparing them with past data. Then, a machine learning model is used to detect abnormal behavior from the webcam footage. The input is the pre-processed clean data, and the output is the anomaly detection data as the analysis result. For example, if the power usage information significantly deviates from past usage patterns, it is flagged as an anomaly.

[1040] Step 4: Detect anomalies and send alerts

[1041] If the server detects an anomaly using the anomaly detection module, the warning module is activated. For example, if the water supply is used continuously for a long period of time, a notification is sent to the user to warn that there may be a water leak. If a fall is detected, the emergency notification system is activated to dispatch an ambulance. The input is the anomaly detection data generated in Step 3, and the output is warning information such as a voice alert, a mobile app notification, or an emergency call.

[1042] Step 5: User preference analysis and recording schedule

[1043] The server uses the user preference recording module to analyze the user's past viewing history and preferences. Based on the data collected here, programs that the user may like are automatically added to the recording reservation list. The input is the user's viewing history data, and the output is a new recording reservation list. For example, if a new drama series is determined to match the user's preferences, the drama will be added to the recording reservation list.

[1044] Step 6: Voice interaction

[1045] When a user speaks into the microphone, the voice data is sent to the server. The server uses a generative AI model to analyze the voice data and generate an appropriate response. The input is the voice input data, and the output is the generated voice response. For example, if you say, "Tell me what the weather is today," the generated response will be, "It's sunny today."

[1046] Step 7: Remote monitoring and operation

[1047] Building managers and family members access a dedicated remote monitoring interface from a terminal. The server monitors the situation inside the home in real time and provides operational information. The input is an access request to the remote monitoring interface, and the output is real-time monitoring data and an operational interface. For example, it is possible to display surveillance footage in real time and remotely turn lights and gas on and off.

[1048] (Application example 1)

[1049] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] Conventional security systems are limited to monitoring electricity, water, and gas usage information and webcam footage individually, making it difficult to detect abnormalities in real time and respond quickly. Furthermore, they lack functionality based on user preferences and voice interaction, leaving a need for greater convenience. Rapid response in emergencies is especially essential for elderly users and those living alone, and current systems are unable to provide sufficient support.

[1051] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1052] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing a warning when an abnormality is detected, means for recording programs according to the user's preferences, means for communicating with the user via voice, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, and means for detecting activity using a webcam and notifying of an abnormality when activity is detected, thereby improving user safety and comfort.

[1053] "Power usage information" is information for acquiring and analyzing power usage data in real time.

[1054] "Water usage information" refers to information for acquiring and analyzing water usage data in real time.

[1055] "Gas usage information" refers to information for acquiring and analyzing gas usage data in real time.

[1056] "Webcam footage" refers to video data captured in real time through a webcam.

[1057] "Analysis" refers to the analytical processing performed to detect abnormalities based on collected data.

[1058] "Issuing a warning" means notifying the user when an abnormality is detected.

[1059] "Recording a program according to the user's tastes and preferences" means automatically recording a program based on the user's past viewing history and interests.

[1060] "Voice interaction" refers to communication between a user and an interface via voice.

[1061] A "front door sensor" is a device that detects whether the front door is open or closed.

[1062] An "interface for remote monitoring and operation" is an interface that allows the system to be monitored and operated from a remote location via a communication means.

[1063] "Motion detection" is the analysis of webcam footage to detect movement within the footage.

[1064] "Notifying an abnormality" means informing the user, administrator, etc. of a detected abnormality.

[1065] As an embodiment of the present invention, a system for collecting and analyzing electricity usage information, water usage information, gas usage information and web camera images in real time to detect abnormalities will be specifically described.

[1066] System configuration

[1067] This system consists of the following main components:

[1068] 1. Data Collection Module

[1069] 2. Data Analysis Module

[1070] 3. Anomaly Detection Module

[1071] 4. Warning Module

[1072] 5. User preference recording module

[1073] 6. Voice Dialogue Module

[1074] 7. Remote monitoring interface

[1075] 8. Motion Detection Module

[1076] Collection of electricity, water and gas usage information

[1077] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building, allowing the data collection module to centrally manage various types of usage information.

[1078] Webcam footage collection

[1079] The server captures real-time video footage from inside the home via a webcam. This is especially important for elderly people and those living alone, as video data is crucial for rapid response in emergencies.

[1080] Data analysis and anomaly detection

[1081] The server analyzes collected electricity, water, and gas usage information, as well as webcam footage, in real time. The data analysis module compares past and current data to detect signs of abnormalities with high accuracy. Webcam footage is also analyzed using machine learning technology to detect abnormalities such as the user falling.

[1082] Response after detecting an anomaly

[1083] The server issues an alert if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance. The motion detection module analyzes webcam footage and similarly notifies the user if it detects movement.

[1084] Providing features based on user preferences

[1085] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[1086] Voice interaction with the user

[1087] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[1088] Remote Monitoring and Operation

[1089] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[1090] Specific examples

[1091] Example 1: Power anomaly detection and notification

[1092] The server analyzes power usage information in real time, and if abnormally high power usage is detected for more than two consecutive hours, the system immediately notifies the user via voice alerts and push notifications on the mobile app.

[1093] Example 2: Detecting falls among elderly people and calling an emergency services

[1094] If the server analyzes the webcam footage and detects that an elderly person is falling to the floor, the emergency call system will automatically dispatch an ambulance and notify their family members.

[1095] Example 3: Recording programs based on preferences

[1096] The server analyzes the user's viewing history, and if it determines that a new drama series matches the user's taste, it automatically adds the drama to the recording reservation list, so the user does not have to worry about missing it.

[1097] Example prompt sentence:

[1098] Please create an app for an anomaly detection system. It will analyze webcam footage in real time and send a notification to a smartphone if an anomaly is detected. Please write pseudocode in Python.

[1099]

[1100] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1101] Step 1:

[1102] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building. This information is input into the data collection module and centrally managed. The specific operation of data collection is to periodically obtain the latest usage data from each smart meter and send it to the server. The input is smart meter data, and the output is integrated usage information data.

[1103] Step 2:

[1104] The server collects real-time video from inside the home through a webcam. This video data is stored appropriately by the data collection module and used for analysis. The webcam captures video data at regular intervals and sends it to the server. The input is the video data from the webcam, and the output is the stored video data.

[1105] Step 3:

[1106] The server inputs the collected electricity, water, and gas usage information, as well as webcam footage, into a data analysis module for real-time analysis. The data analysis module compares past and current usage data and applies machine learning algorithms to detect signs of anomalies. For example, it detects an abnormal increase in electricity usage information. The input is past and current usage data and video data, and the output is the analysis results.

[1107] Step 4:

[1108] If the server detects an anomaly based on the analysis results, it issues a warning. The warning module notifies the user of the anomaly through voice alerts, mobile app push notifications, email notifications, etc. Specifically, it generates an appropriate warning message according to the detected anomaly and sends it through the notification means. The input is the analysis result, and the output is the warning message.

[1109] Step 5:

[1110] Based on the user's preferences, the server analyzes the viewing history and automatically schedules recordings of programs that the user may be interested in. The user preference recording module uses a generative AI model based on the viewing history data to predict programs that the user is likely to watch in the future. The input is the user's viewing history data, and the output is a recording schedule list.

[1111] Step 6:

[1112] When a user speaks into the microphone, the server uses a generative AI model to analyze the voice and generate an appropriate response. Specifically, the voice dialogue module converts the voice input into text, and the generative AI model generates an answer based on that text. The input is the user's voice data, and the output is the response voice data.

[1113] Step 7:

[1114] When the door sensor detects the user's return home, the server plays a voice message. The door sensor sends the detection data as input, and the voice interaction module generates and plays an appropriate voice message. The input is the door sensor data, and the output is the voice message to be played.

[1115] Step 8:

[1116] Building managers and family members access a dedicated remote monitoring interface from a terminal, and the server provides an interface that allows them to monitor and control the situation inside the home in real time. This allows for quick response when an abnormality occurs or when regular checks are required. The input is an access request from the monitoring interface, and the output is real-time status information.

[1117] Step 9:

[1118] The server analyzes webcam footage to detect movement. The motion detection module detects specific movements and treats them as an anomaly, issuing a notification if necessary. Specifically, it analyzes changes between video frames to determine whether there is a consistent movement. The input is the webcam video data, and the output is an anomaly notification.

[1119] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1120] As a form for implementing the present invention, we will specifically explain a system that combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities, with an emotion engine that recognizes user emotions.

[1121] System configuration

[1122] This system consists of the following main components:

[1123] 1. Data Collection Module

[1124] 2. Data Analysis Module

[1125] 3. Anomaly Detection Module

[1126] 4. Warning Module

[1127] 5. User preference recording module

[1128] 6. Voice Dialogue Module

[1129] 7. Emotion Engine Module

[1130] 8. Remote monitoring interface

[1131] Collection and analysis of electricity, water and gas usage information

[1132] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building. This allows the data collection module to centrally manage various usage information. The server analyzes the collected data and compares it with past data to detect anomalies.

[1133] Webcam footage collection and analysis

[1134] The server captures real-time video footage from inside the home via a webcam, which is then analyzed using machine learning technology to detect abnormal behavior such as falls.

[1135] Emotion recognition by emotion engine

[1136] The server captures the user's audio and video data and analyzes it with an emotion engine module, which uses machine learning models to analyze the user's facial expressions, tone of voice, and other factors to recognize the user's emotions in real time.

[1137] Response after detecting an anomaly

[1138] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[1139] Emotion-based responses

[1140] The server generates appropriate voice dialogue based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, it can recommend relaxing music. It can also automatically adjust the brightness of lights and music according to the user's emotions.

[1141] Providing features based on user preferences

[1142] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[1143] Voice interaction with the user

[1144] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[1145] Remote Monitoring and Operation

[1146] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[1147] Specific examples

[1148] Example 1: Program recommendation using emotion recognition

[1149] If the server recognizes through its emotion engine that the user is feeling stressed, it will recommend relaxing movies and TV shows that suit the user's preferences and automatically record them.

[1150] Example 2: Adjusting the environment based on emotions

[1151] If the server recognizes through its emotion engine that the user is feeling tired, it will dim the lights in the room and play relaxing music.

[1152] Example 3: Emergency response with fall detection and emotion recognition

[1153] If the server detects a fall and the emotion engine recognizes pain or confusion from the user's facial expression, it will immediately activate the emergency call system and dispatch an ambulance.

[1154] In this way, the present invention makes full use of advanced technology to achieve safety, comfort, and optimal response according to the user's emotions.

[1155] The processing flow will be explained below.

[1156] Step 1:

[1157] The server obtains electricity usage information from the smart meter every minute via an API.

[1158] Step 2:

[1159] The server obtains water usage information from the water meter every 10 minutes via an API.

[1160] Step 3:

[1161] The server obtains gas usage information from the gas meter every 15 minutes via an API.

[1162] Step 4:

[1163] The server receives and temporarily stores the video stream from the webcam in real time.

[1164] Step 5:

[1165] The server analyzes the collected power usage information and compares it with data from the past 24 hours to detect abnormally high power usage.

[1166] Step 6:

[1167] The server analyzes water usage information and detects abnormalities when water is used continuously for long periods of time.

[1168] Step 7:

[1169] The server analyzes gas usage information and monitors sudden fluctuations in gas usage in real time.

[1170] Step 8:

[1171] The server analyzes webcam footage using machine learning models to detect abnormal behavior such as the user falling.

[1172] Step 9:

[1173] The server captures the user's voice in real time and analyzes it with an emotion engine to recognize emotions.

[1174] Step 10:

[1175] The server acquires the user's facial expression data from a webcam and analyzes it with an emotion engine to recognize emotions.

[1176] Step 11:

[1177] The server generates appropriate voice dialogue for the user based on the recognized emotion. For example, if the user looks sad, the server might say, "Is there something I can help you with?"

[1178] Step 12:

[1179] Based on the recognized emotions, the server recommends TV programs and movies that match the user's preferences and automatically records them.

[1180] Step 13:

[1181] The server will send a warning alert to the user if an abnormality is detected. For example, if water continues to flow for a long period of time, it will immediately notify the user that there may be a water leak.

[1182] Step 14:

[1183] The server will activate the emergency call system and dispatch an ambulance if a fall is detected, and will also initiate emergency response if the emotion engine detects emotions such as pain or confusion.

[1184] Step 15:

[1185] If the server detects an emotion requiring relaxation through its emotion engine, it will dim the lights slightly and play relaxing music.

[1186] Step 16:

[1187] The server detects that the user has returned home via the entrance sensor and plays a voice message saying "Welcome home."

[1188] Step 17:

[1189] Administrators and family members can access a dedicated remote monitoring interface from the device and check various data in real time.

[1190] Step 18:

[1191] Remotely control electricity, water, and gas usage as needed from your device.

[1192] By performing detailed processing step by step in this way, the safety and comfort of the entire system and optimal responses according to the user's emotions are ensured.

[1193] Example 2

[1194] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1195] Conventional monitoring systems collect electricity, water, and gas usage information and webcam footage, but are limited in their ability to comprehensively analyze this information and detect abnormalities. They also lack the means to analyze a user's emotional state in real time and take appropriate action based on the abnormality or emotion. Furthermore, they lack the functionality to recommend content based on the user's preferences and generate voice dialogue. The present invention aims to solve these problems and provide a safe and comfortable living environment.

[1196] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1197] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing an alert when an abnormality is detected, means for recording programs according to a user's preferences, means for engaging in voice conversation with the user, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, means for analyzing the user's emotions, means for adjusting the environment based on the user's emotions, and means for generating voice dialogue using a generative AI model. This enables the server to comprehensively analyze electricity, water, and gas usage information and video data, detect abnormalities, and take appropriate measures, as well as adjust the environment, recommend content, and engage in voice dialogue according to the user's emotional state.

[1198] "Electricity usage information" refers to data on the usage of electricity within homes and buildings.

[1199] "Water usage information" refers to data such as the amount of water used within a home or building and the timing of use.

[1200] "Gas usage information" refers to data regarding the amount and timing of gas usage within homes and buildings.

[1201] "Webcam Footage" refers to video footage data collected through a webcam.

[1202] "Means for detecting anomalies" refers to algorithms or analytical systems that identify anomalies by comparing them with typical usage and historical data.

[1203] "Means of issuing an alert" refers to a system or function that sends a notification to a user or administrator when an abnormality is detected.

[1204] "Means for recording programs according to the user's tastes and preferences" refers to a system that automatically records highly relevant programs based on the user's past viewing history and preferences.

[1205] "Means of interacting with the user via voice" refers to a system that uses speech recognition and generative AI models to provide appropriate responses to user utterances.

[1206] "Means for detecting the user's return home using a front door sensor and playing a voice message" refers to a system that detects the user's return home based on information obtained from a front door sensor and plays a pre-set voice message.

[1207] "Means for providing an interface for remote monitoring and operation" refers to an interface that allows building managers and family members to monitor the situation within the home in real time via the Internet and perform necessary operations.

[1208] "Means for analyzing user emotions" refers to a system that analyzes the user's facial expressions and voice data and recognizes their emotional state in real time.

[1209] "Means for adjusting the environment based on the user's emotions" refers to a system that automatically adjusts environmental settings such as room lighting and music according to the user's emotional state.

[1210] "Means for generating a voice dialogue using a generative AI model" refers to a system that uses an AI model to generate a voice dialogue with a user in real time.

[1211] This system combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect anomalies, with an emotion engine that recognizes user emotions. This system consists of the following main components:

[1212] System configuration

[1213] 1. Data Collection Module

[1214] 2. Data Analysis Module

[1215] 3. Anomaly Detection Module

[1216] 4. Warning Module

[1217] 5. User preference recording module

[1218] 6. Voice Dialogue Module

[1219] 7. Emotion Engine Module

[1220] 8. Remote monitoring interface

[1221] Collection and analysis of electricity, water and gas usage information

[1222] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building and manages it centrally. This allows the data collection module to periodically update the usage information. The collected data is stored in a database and processed by the data analysis module. For example, an algorithm is applied to compare the data with past data to detect anomalies in usage.

[1223] Webcam footage collection and analysis

[1224] The server captures real-time video footage from inside the home via a webcam. This video data is analyzed using machine learning technology to detect abnormal behavior, such as falls. If an abnormality is detected, the anomaly detection module immediately issues an alert and, if necessary, activates the emergency call system.

[1225] Emotion recognition by emotion engine

[1226] The server acquires the user's audio and video data and analyzes it with an emotion engine module. This emotion engine uses machine learning models to analyze the user's facial expressions, tone of voice, and other factors to recognize their emotional state in real time.

[1227] Response after detecting an anomaly

[1228] The server has a means to issue a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. Also, if a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[1229] Emotion-based responses

[1230] The server generates appropriate voice dialogue based on the user's emotions recognized by the emotion engine. If the user is feeling stressed, it can recommend relaxing music and adjust the lighting in the room. Other features include recommending entertainment content based on the user's emotional state.

[1231] Providing features based on user preferences

[1232] The server analyzes the user's past viewing history and preferences, and has the function of recommending and recording content that the user may be interested in. The recording reservation list is updated in real time to optimize the entertainment experience.

[1233] Voice interaction with the user

[1234] The server uses a generative AI model to converse with the user via voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated in real time. For example, it is possible to have a conversation about everyday information such as the day's schedule, weather, and news.

[1235] Remote Monitoring and Operation

[1236] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to respond quickly and appropriately when an abnormality occurs or regular checks are required.

[1237] Examples and prompts

[1238] Example 1: Program recommendation using emotion recognition

[1239] If the server recognizes through its emotion engine that the user is feeling stressed, it will recommend relaxing movies and TV shows that suit the user's preferences and automatically record them.

[1240] Prompt: "Explain how an emotion recognition engine can recommend relaxing content when it detects stress."

[1241] Example 2: Adjusting the environment based on emotions

[1242] When the server's emotion engine recognizes that the user is feeling tired, it dims the lights in the room and plays relaxing music.

[1243] Prompt: "Describe how you can automatically adjust the environment in a room based on your emotional state."

[1244] Example 3: Emergency response with fall detection and emotion recognition

[1245] If the server detects a fall and the emotion engine recognizes pain or confusion from the user's facial expression, the emergency call system will immediately be activated and an ambulance will be dispatched.

[1246] Prompt: "Please describe your emergency response process when you detect a client has fallen."

[1247] In this way, the present invention utilizes advanced technology to provide optimal responses based on the user's safety, comfort, and emotions.

[1248] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1249] Step 1: Data collection

[1250] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building. This data is input into a data collection module and stored centrally in a database. Specifically, the data sent from the smart meters is obtained using the HTTP protocol and stored as is in the database.

[1251] Input: Real-time data from smart meters

[1252] Output: Electricity, water, and gas usage information stored in a database

[1253] Step 2: Data analysis

[1254] The server analyzes the electricity, water, and gas usage information stored in the database. This involves comparing usage with past data and using algorithms to detect patterns. Specifically, it compares data from the past month and performs statistical analysis to identify sudden increases or decreases in usage.

[1255] Input: Historical electricity, water, and gas usage information stored in a database

[1256] Output: Statistical data for anomaly detection

[1257] Step 3: Anomaly detection

[1258] The server uses the statistical data obtained from the data analysis module to detect anomalies. If an anomaly is detected, the information is input into the anomaly detection module, which generates a warning message. Specifically, if an abnormal pattern is discovered, an alert flag is set and sent to the user notification system.

[1259] Input: Statistical analysis data

[1260] Output: Anomaly detection alert flag and warning message

[1261] Step 4: Collect and analyze webcam footage

[1262] The server collects real-time video data from the webcam and analyzes it using machine learning technology. Specifically, the video analysis algorithm detects abnormal behavior such as falls and issues an alert if necessary.

[1263] Input: Real-time video data from a webcam

[1264] Output: Detected abnormal behavior and warning message

[1265] Step 5: Emotion Recognition

[1266] The server inputs the user's audio and video data into the emotion engine module and analyzes their emotional state. This allows the system to recognize emotions in real time based on the user's facial expressions and tone of voice. Specifically, the emotion engine analyzes the user's audio and video data and generates emotion tags.

[1267] Input: Audio and video data

[1268] Output: Emotion tag

[1269] Step 6: Dealing with abnormalities and emotions

[1270] The server responds appropriately based on the abnormality and emotional state. If an abnormality occurs, the emergency notification system is activated, and depending on the emotional state, the response may include adjusting the music or lighting. Specifically, if an abnormality is detected, a warning message is sent, and if the emotional state is identified, a music playback API is called to play a playlist.

[1271] Input: Anomaly detection alert flag or sentiment tag

[1272] Output: Emergency call, music playback, lighting control

[1273] Step 7: User Preference Management

[1274] The server analyzes the user's past viewing history and preference data, and recommends and records content that the user may be interested in. Specifically, it queries the viewing history database, and the recommendation engine suggests appropriate content.

[1275] Input: Viewing history database

[1276] Output: Recommended content

[1277] Step 8: Voice interaction generation

[1278] The server uses a voice interaction module to communicate with the user via voice. The generative AI model analyzes what the user says and generates an appropriate response. Specifically, the server converts the user's voice input into text, inputs that text into the generative AI model to generate a response, and then converts that response into voice.

[1279] Input: User voice input

[1280] Output: Voice reply

[1281] Step 9: Remote monitoring and operation

[1282] The terminal allows building managers and family members to access the remote monitoring interface and monitor and control the situation inside the home in real time.Specifically, they log in to the remote monitoring interface, check camera footage and smart meter data inside the home, and remotely change lighting and air conditioning settings as needed.

[1283] Input: Real-time monitoring data

[1284] Output: Operation instructions

[1285] (Application example 2)

[1286] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1287] Conventional anomaly detection systems monitor usage information for electricity, water, gas, etc. in real time, and have the ability to detect abnormalities and issue alarms, but they are unable to grasp the emotional state of the user and take appropriate action.In addition, there are limitations to the means of quickly notifying households and administrators when an abnormality occurs, so further improvements in safety and comfort are needed.

[1288] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1289] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing a warning when an abnormality is detected, means for recording programs according to a user's preferences, means for audio communication with the user, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, means for recognizing a user's emotion, means for providing warnings and advice based on the user's emotional state, means for detecting abnormalities based on the electricity, water, and gas usage information and sending an electronic message, and means for making an emergency call based on the webcam footage. This enables not only a rapid response when an abnormality is detected but also the provision of advanced warnings and advice based on the user's emotional state.

[1290] - "Electricity usage information" means data indicating the amount of electricity consumed within a home or facility.

[1291] "Water usage information" is data that indicates the amount of water used within a home or facility.

[1292] "Gas usage information" means data indicating the amount of gas used within a home or facility.

[1293] "Webcam footage" refers to real-time video data captured using a webcam.

[1294] "Means for detecting anomalies" refers to analytical means for identifying unusual conditions or behaviors from collected data.

[1295] "Means for issuing warnings" refers to notification means for alerting users and administrators when an abnormality is detected.

[1296] The "means for recording programs according to the user's tastes and preferences" is a means for automatically recording programs and video content that the user likes.

[1297] "Means for communicating with the user by voice" refers to an interface that responds to and communicates with the user by voice.

[1298] The "entrance sensor" is a sensor that detects whether the entrance is open or closed.

[1299] The "means for playing back a voice message" is a means for playing back a voice message when the user returns home.

[1300] "Interface for remote monitoring and operation" refers to an interface that allows a manager or family member in a remote location to monitor and operate the situation within the home in real time.

[1301] "Means for recognizing user emotions" refers to means for identifying the user's emotional state by analyzing the user's facial expressions and voice.

[1302] The "means for providing warnings and advice based on emotional state" refers to means for providing appropriate warnings and advice to a user in accordance with the recognized emotional state.

[1303] "Means for sending electronic messages" refers to means for sending electronic messages to users or administrators when an abnormality is detected.

[1304] "Means for making an emergency call" refers to a means for making an emergency call when an abnormality that poses a threat to human life is detected based on webcam footage.

[1305] As a form for implementing the present invention, we will explain a specific embodiment of a security service that combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities, with an emotion engine that recognizes user emotions.

[1306] System configuration

[1307] This system consists of the following main components:

[1308] 1. Data Collection Module

[1309] The server uses smart meters and web cameras to obtain electricity usage information, water usage information, gas usage information, and real-time images from inside the home.

[1310] 2. Data Analysis Module

[1311] The server analyzes the collected electricity, water, and gas usage information and webcam footage to detect abnormalities, and identifies them by comparing them with past usage data.

[1312] 3. Anomaly Detection Module

[1313] The server will issue a warning to the user if it detects any abnormalities from the analysis results, such as if the water supply has been running for an extended period of time or if it detects abnormal behavior based on webcam footage.

[1314] 4. Emotion Engine Module

[1315] The server recognizes the user's emotions using webcam footage and audio data acquired from a microphone. This emotion engine uses a machine learning model to analyze facial expressions and tone of voice to grasp the user's emotional state in real time.

[1316] 5. Warning Module

[1317] When an abnormality is detected, the server will send a warning to the user via email or push notification. In addition, if the emotion engine recognizes that the user is feeling stressed or anxious, it will provide the user with appropriate advice.

[1318] 6. Voice Dialogue Module

[1319] The server uses a generative AI model to interact with the user through voice: when the user speaks into the microphone, the content is analyzed and an appropriate response is generated.

[1320] 7. Emotion-based responses

[1321] The server can recommend relaxing music or adjust the lighting in the room based on the user's emotions recognized by the emotion engine.

[1322] As a specific example, if a user is feeling stressed, the system may notify them, "You seem to be feeling stressed. I'll play some music to help you relax," and play a relaxing song from the user's music library.

[1323] Examples of prompts to be input to a generative AI model include, "The user's emotions have changed. Please provide a response based on their latest emotional state," and "An anomaly has been detected. Please suggest an appropriate response."

[1324] 8. Remote monitoring interface

[1325] Administrators and family members can access a dedicated remote monitoring interface from a terminal and monitor and operate the system in real time, allowing for immediate response when an abnormality occurs or when regular checks are required.

[1326] This system uses hardware such as smartphones, webcams, and smart meters, and software such as Python, OpenCV, and Keras (with TensorFlow backend). The server collects, analyzes, and processes data from these hardware devices to improve user safety and comfort.

[1327] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1328] Step 1:

[1329] Data collection

[1330] The server collects electricity usage information, water usage information, gas usage information, and webcam footage in real time from smart meters and webcams. The input is data from each sensor and device, and the output is real-time data that is stored in a database on the server.

[1331] Step 2:

[1332] Data analysis

[1333] The server compares collected electricity, water, and gas usage information with past data. It also performs face detection and abnormal behavior detection on webcam footage. The input is real-time and past data, and the output is an alert indicating whether or not an abnormality has occurred.

[1334] Step 3:

[1335] Anomaly detection

[1336] The server detects anomalies from the analysis results. If an anomaly is detected, it generates a warning message. The input is the analysis results, and the output is the warning message.

[1337] Step 4:

[1338] emotion recognition

[1339] The server uses the webcam video and microphone audio data to recognize the user's emotions with an emotion engine. The input is real-time video and audio data, and the output is data representing the user's emotional state.

[1340] Step 5:

[1341] Warning

[1342] If an anomaly is detected or if the emotion engine recognizes the user's stress or anxiety, the server issues a warning to the user via email, push notification, etc. The input is the anomaly detection result and emotion recognition result, and the output is email or push notification.

[1343] Step 6:

[1344] Voice dialogue

[1345] The server analyzes what the user says into the microphone using a generative AI model and generates an appropriate response. The input is voice data, and the output is the generated response.

[1346] Step 7:

[1347] Emotion-based responses

[1348] The server recommends music suitable for the user and adjusts lighting based on the recognition results of the emotion engine. The input is emotion recognition data, and the output is commands to play music or control lighting.

[1349] Step 8:

[1350] Remote Monitoring

[1351] Administrators and family members can access a dedicated remote monitoring interface from a terminal to monitor and operate the situation in real time. The input is monitoring data and operation instructions, and the output is the monitoring results and the executed operations.

[1352] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1353] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1354] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1355] [Fourth embodiment]

[1356] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1357] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1358] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1359] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1360] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1361] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1362] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1363] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1364] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1365] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1366] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1367] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1368] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1369] As an embodiment of the present invention, a system for collecting and analyzing electricity usage information, water usage information, gas usage information and web camera images in real time to detect abnormalities will be specifically described.

[1370] System configuration

[1371] This system consists of the following main components:

[1372] 1. Data Collection Module

[1373] 2. Data Analysis Module

[1374] 3. Anomaly Detection Module

[1375] 4. Warning Module

[1376] 5. User preference recording module

[1377] 6. Voice Dialogue Module

[1378] 7. Remote monitoring interface

[1379] Collection of electricity, water and gas usage information

[1380] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building, allowing the data collection module to centrally manage various types of usage information.

[1381] Webcam footage collection

[1382] The server captures real-time video footage from inside the home via a webcam. This is particularly important for elderly people and those living alone, as video data is crucial for rapid response in emergencies.

[1383] Data analysis and anomaly detection

[1384] The server analyzes collected electricity, water, and gas usage information, as well as webcam footage, in real time. The data analysis module compares past and current data to detect signs of abnormalities with high accuracy. Webcam footage is also analyzed using machine learning technology to detect abnormalities such as the user falling.

[1385] Response after detecting an anomaly

[1386] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[1387] Providing features based on user preferences

[1388] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[1389] Voice interaction with the user

[1390] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[1391] Remote Monitoring and Operation

[1392] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[1393] Specific examples

[1394] Example 1: Power anomaly detection and notification

[1395] The server analyzes power usage information in real time, and if abnormally high power usage is detected for more than two consecutive hours, the system immediately notifies the user via voice alerts and push notifications on the mobile app.

[1396] Example 2: Detecting falls among elderly people and calling an emergency services

[1397] If the server analyzes the webcam footage and detects that an elderly person is falling to the floor, the emergency call system will automatically dispatch an ambulance and notify their family members.

[1398] Example 3: Recording programs based on preferences

[1399] The server analyzes the user's viewing history, and if it determines that a new drama series matches the user's taste, it automatically adds the drama to the recording reservation list, so the user does not have to worry about missing it.

[1400] Thus, the invention utilizes advanced technology to provide safety, comfort, and reduced loneliness.

[1401] The processing flow will be explained below.

[1402] Step 1:

[1403] The server obtains electricity usage information from the smart meter every minute via an API.

[1404] Step 2:

[1405] The server obtains water usage information from the water meter every 10 minutes via an API.

[1406] Step 3:

[1407] The server obtains gas usage information from the gas meter every 15 minutes via an API.

[1408] Step 4:

[1409] The server receives and temporarily stores the video stream from the webcam in real time.

[1410] Step 5:

[1411] The server analyzes the collected power usage information and compares it with data from the past 24 hours to detect abnormally high power usage.

[1412] Step 6:

[1413] The server analyzes water usage information and detects abnormalities when water is used continuously for long periods of time.

[1414] Step 7:

[1415] The server analyzes gas usage information and monitors sudden fluctuations in gas usage in real time.

[1416] Step 8:

[1417] The server analyzes webcam footage using machine learning models to detect abnormal behavior such as the user falling.

[1418] Step 9:

[1419] If the server detects any abnormality, it will immediately send a warning alert to the user.

[1420] Step 10:

[1421] If a fall is detected, the server activates an emergency call system and dispatches an ambulance.

[1422] Step 11:

[1423] The server analyzes the user's viewing history and generates a list of programs to be automatically recorded based on the user's preferences.

[1424] Step 12:

[1425] The server detects that the user has returned home via the entrance sensor and plays a voice message saying "Welcome home."

[1426] Step 13:

[1427] When the user speaks into the microphone, the server converts the voice data into text and generates an appropriate response that is played back aloud.

[1428] Step 14:

[1429] Administrators and family members can access a dedicated remote monitoring interface from the device and check various data in real time.

[1430] Step 15:

[1431] Remotely control electricity, water, and gas usage as needed from your device.

[1432] By performing detailed processing at each step in this way, safety and comfort of the entire system are ensured.

[1433] Example 1

[1434] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1435] In modern society, there is a need for systems that allow the elderly and people living alone to quickly respond to abnormalities and emergencies in their daily lives. Furthermore, there is a common need for systems that provide entertainment based on users' hobbies and preferences, streamline daily communication, and enable remote monitoring and operation. Current systems have difficulty providing these functions in an integrated manner, making it essential to provide a comprehensive system that significantly improves safety and convenience.

[1436] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1437] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage using a data analysis module to detect abnormalities, means for issuing an alert when an abnormality is detected, means for recording programs using a user preference analysis module, means for communicating with users via voice using a generative AI model, and means for monitoring and operating through a remote monitoring interface. This enables quick response in emergencies, optimization of entertainment, and improvement of daily communication efficiency while improving the safety and comfort of residents' lives.

[1438] 1. "Electricity usage information" refers to data on the amount and usage patterns of electricity consumed by facilities such as homes and buildings.

[1439] 2. "Water usage information" means data on the amount and patterns of water consumed by households, buildings, and other facilities.

[1440] 3. "Gas usage information" means data relating to the amount and usage patterns of gas consumed in homes, buildings, and other facilities.

[1441] 4. "Webcam footage" means real-time visual data obtained from cameras installed within a home.

[1442] 5. "Data Analysis Module" means an integrated system of software and hardware used to analyze various collected data and detect patterns and anomalies.

[1443] 6. "Anomaly detection means" means a technology that utilizes a data analysis module to recognize data patterns or irregular behavior that exceed certain thresholds in real time.

[1444] 7. "Means of issuing warnings" refers to communication methods such as voice alerts, push notifications, emails, etc., used to notify users and relevant parties when an abnormality is detected.

[1445] 8. "Preference Analysis Module" is a system that analyzes a user's viewing history and behavioral patterns and provides content and services based on the user's preferences.

[1446] 9. “Generative AI model” means an artificial intelligence technology used to generate natural-sounding voice interactions with users and appropriate responses.

[1447] 10. "Remote monitoring interface" means a user interface for monitoring the situation inside a home or facility in real time from an external terminal and performing necessary operations.

[1448] This invention relates to a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities. This system issues warnings based on the results of abnormality detection, records programs based on user preferences, and provides voice interaction. It also includes an interface for remote monitoring and operation.

[1449] System configuration

[1450] The system consists of the following main components:

[1451] 1. Data Collection Module

[1452] 2. Data Analysis Module

[1453] 3. Anomaly Detection Module

[1454] 4. Warning Module

[1455] 5. User preference recording module

[1456] 6. Voice Dialogue Module

[1457] 7. Remote monitoring interface

[1458] Collection of electricity, water and gas usage information

[1459] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building, allowing the data collection module to centrally manage various types of usage information.

[1460] Webcam footage collection

[1461] The server receives real-time video from web cameras in homes, and this video data is especially important for emergency response for elderly users and those living alone.

[1462] Data analysis and anomaly detection

[1463] The server uses a data analysis module to analyze the collected electricity, water, and gas usage information, as well as webcam footage, in real time. This module compares past data with current data to detect signs of abnormalities. Additionally, the webcam footage is analyzed using machine learning technology to detect abnormalities such as the user falling.

[1464] Response after detecting an anomaly

[1465] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it will notify the user that there may be a water leak. If a fall is detected, the emergency notification system will automatically call an ambulance and notify the user's family.

[1466] Providing features based on user preferences

[1467] The server uses a user preference recording module to analyze the user's past viewing history and preferences, and can automatically record programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[1468] Voice interaction with the user

[1469] The server uses a generative AI model to communicate with the user via voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, in response to a question such as "What's the weather like today?", the server can provide weather information.

[1470] Remote Monitoring and Operation

[1471] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when performing regular checks.

[1472] Prompt Sentence Examples

[1473] "Please tell me my name and what the weather is like today."

[1474] "Can you record a new show similar to the one I've been watching lately?"

[1475] "My home's electricity usage seems higher than usual. Is there a problem?"

[1476] In this way, the system of the present invention aims to improve the safety and comfort of residents' lives by using advanced data collection and analysis technology.

[1477] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1478] Step 1: Data collection

[1479] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building. It also acquires real-time video footage from web cameras installed in homes. The input is data obtained from various smart meters and web cameras, and the output is stored in a database as integrated usage information data and video data.

[1480] Step 2: Initial Data Processing

[1481] The server performs initial processing of the collected data using a data collection module. Specifically, data format conversion and filtering of unnecessary data are performed. The input is the data collected in step 1, and the output is clean data that has been converted into an analyzable format. For example, the units of power usage information can be standardized, and noisy webcam footage can be cleaned up using a noise reduction filter.

[1482] Step 3: Data analysis

[1483] The server analyzes the pre-processed data using a data analysis module. First, an algorithm is applied to detect anomalies by comparing them with past data. Then, a machine learning model is used to detect abnormal behavior from the webcam footage. The input is the pre-processed clean data, and the output is the anomaly detection data as the analysis result. For example, if the power usage information significantly deviates from past usage patterns, it is flagged as an anomaly.

[1484] Step 4: Detect anomalies and send alerts

[1485] If the server detects an anomaly using the anomaly detection module, the warning module is activated. For example, if the water supply is used continuously for a long period of time, a notification is sent to the user to warn that there may be a water leak. If a fall is detected, the emergency notification system is activated to dispatch an ambulance. The input is the anomaly detection data generated in Step 3, and the output is warning information such as a voice alert, a mobile app notification, or an emergency call.

[1486] Step 5: User preference analysis and recording schedule

[1487] The server uses the user preference recording module to analyze the user's past viewing history and preferences. Based on the data collected here, programs that the user may like are automatically added to the recording reservation list. The input is the user's viewing history data, and the output is a new recording reservation list. For example, if a new drama series is determined to match the user's preferences, the drama will be added to the recording reservation list.

[1488] Step 6: Voice interaction

[1489] When a user speaks into the microphone, the voice data is sent to the server. The server uses a generative AI model to analyze the voice data and generate an appropriate response. The input is the voice input data, and the output is the generated voice response. For example, if you say, "Tell me what the weather is today," the generated response will be, "It's sunny today."

[1490] Step 7: Remote monitoring and operation

[1491] Building managers and family members access a dedicated remote monitoring interface from a terminal. The server monitors the situation inside the home in real time and provides operational information. The input is an access request to the remote monitoring interface, and the output is real-time monitoring data and an operational interface. For example, it is possible to display surveillance footage in real time and remotely turn lights and gas on and off.

[1492] (Application example 1)

[1493] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1494] Conventional security systems are limited to monitoring electricity, water, and gas usage information and webcam footage individually, making it difficult to detect abnormalities in real time and respond quickly. Furthermore, they lack functionality based on user preferences and voice interaction, leaving a need for greater convenience. Rapid response in emergencies is especially essential for elderly users and those living alone, and current systems are unable to provide sufficient support.

[1495] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1496] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing a warning when an abnormality is detected, means for recording programs according to the user's preferences, means for communicating with the user via voice, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, and means for detecting activity using a webcam and notifying of an abnormality when activity is detected, thereby improving user safety and comfort.

[1497] "Power usage information" is information for acquiring and analyzing power usage data in real time.

[1498] "Water usage information" refers to information for acquiring and analyzing water usage data in real time.

[1499] "Gas usage information" refers to information for acquiring and analyzing gas usage data in real time.

[1500] "Webcam footage" refers to video data captured in real time through a webcam.

[1501] "Analysis" refers to the analytical processing performed to detect abnormalities based on collected data.

[1502] "Issuing a warning" means notifying the user when an abnormality is detected.

[1503] "Recording a program according to the user's tastes and preferences" means automatically recording a program based on the user's past viewing history and interests.

[1504] "Voice interaction" refers to communication between a user and an interface via voice.

[1505] A "front door sensor" is a device that detects whether the front door is open or closed.

[1506] An "interface for remote monitoring and operation" is an interface that allows the system to be monitored and operated from a remote location via a communication means.

[1507] "Motion detection" is the analysis of webcam footage to detect movement within the footage.

[1508] "Notifying an abnormality" means informing the user, administrator, etc. of a detected abnormality.

[1509] As an embodiment of the present invention, a system for collecting and analyzing electricity usage information, water usage information, gas usage information and web camera images in real time to detect abnormalities will be specifically described.

[1510] System configuration

[1511] This system consists of the following main components:

[1512] 1. Data Collection Module

[1513] 2. Data Analysis Module

[1514] 3. Anomaly Detection Module

[1515] 4. Warning Module

[1516] 5. User preference recording module

[1517] 6. Voice Dialogue Module

[1518] 7. Remote monitoring interface

[1519] 8. Motion Detection Module

[1520] Collection of electricity, water and gas usage information

[1521] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building, allowing the data collection module to centrally manage various types of usage information.

[1522] Webcam footage collection

[1523] The server captures real-time video footage from inside the home via a webcam. This is especially important for elderly people and those living alone, as video data is crucial for rapid response in emergencies.

[1524] Data analysis and anomaly detection

[1525] The server analyzes collected electricity, water, and gas usage information, as well as webcam footage, in real time. The data analysis module compares past and current data to detect signs of abnormalities with high accuracy. Webcam footage is also analyzed using machine learning technology to detect abnormalities such as the user falling.

[1526] Response after detecting an anomaly

[1527] The server issues an alert if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance. The motion detection module analyzes webcam footage and similarly notifies the user if it detects movement.

[1528] Providing features based on user preferences

[1529] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[1530] Voice interaction with the user

[1531] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[1532] Remote Monitoring and Operation

[1533] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[1534] Specific examples

[1535] Example 1: Power anomaly detection and notification

[1536] The server analyzes power usage information in real time, and if abnormally high power usage is detected for more than two consecutive hours, the system immediately notifies the user via voice alerts and push notifications on the mobile app.

[1537] Example 2: Detecting falls among elderly people and calling an emergency services

[1538] If the server analyzes the webcam footage and detects that an elderly person is falling to the floor, the emergency call system will automatically dispatch an ambulance and notify their family members.

[1539] Example 3: Recording programs based on preferences

[1540] The server analyzes the user's viewing history, and if it determines that a new drama series matches the user's taste, it automatically adds the drama to the recording reservation list, so the user does not have to worry about missing it.

[1541] Example prompt sentence:

[1542] Please create an app for an anomaly detection system. It will analyze webcam footage in real time and send a notification to a smartphone if an anomaly is detected. Please write pseudocode in Python.

[1543]

[1544] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1545] Step 1:

[1546] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building. This information is input into the data collection module and centrally managed. The specific operation of data collection is to periodically obtain the latest usage data from each smart meter and send it to the server. The input is smart meter data, and the output is integrated usage information data.

[1547] Step 2:

[1548] The server collects real-time video from inside the home through a webcam. This video data is stored appropriately by the data collection module and used for analysis. The webcam captures video data at regular intervals and sends it to the server. The input is the video data from the webcam, and the output is the stored video data.

[1549] Step 3:

[1550] The server inputs the collected electricity, water, and gas usage information, as well as webcam footage, into a data analysis module for real-time analysis. The data analysis module compares past and current usage data and applies machine learning algorithms to detect signs of anomalies. For example, it detects an abnormal increase in electricity usage information. The input is past and current usage data and video data, and the output is the analysis results.

[1551] Step 4:

[1552] If the server detects an anomaly based on the analysis results, it issues a warning. The warning module notifies the user of the anomaly through voice alerts, mobile app push notifications, email notifications, etc. Specifically, it generates an appropriate warning message according to the detected anomaly and sends it through the notification means. The input is the analysis result, and the output is the warning message.

[1553] Step 5:

[1554] Based on the user's preferences, the server analyzes the viewing history and automatically schedules recordings of programs that the user may be interested in. The user preference recording module uses a generative AI model based on the viewing history data to predict programs that the user is likely to watch in the future. The input is the user's viewing history data, and the output is a recording schedule list.

[1555] Step 6:

[1556] When a user speaks into the microphone, the server uses a generative AI model to analyze the voice and generate an appropriate response. Specifically, the voice dialogue module converts the voice input into text, and the generative AI model generates an answer based on that text. The input is the user's voice data, and the output is the response voice data.

[1557] Step 7:

[1558] When the door sensor detects the user's return home, the server plays a voice message. The door sensor sends the detection data as input, and the voice interaction module generates and plays an appropriate voice message. The input is the door sensor data, and the output is the voice message to be played.

[1559] Step 8:

[1560] Building managers and family members access a dedicated remote monitoring interface from a terminal, and the server provides an interface that allows them to monitor and control the situation inside the home in real time. This allows for quick response when an abnormality occurs or when regular checks are required. The input is an access request from the monitoring interface, and the output is real-time status information.

[1561] Step 9:

[1562] The server analyzes webcam footage to detect movement. The motion detection module detects specific movements and treats them as an anomaly, issuing a notification if necessary. Specifically, it analyzes changes between video frames to determine whether there is a consistent movement. The input is the webcam video data, and the output is an anomaly notification.

[1563] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1564] As a form for implementing the present invention, we will specifically explain a system that combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities, with an emotion engine that recognizes user emotions.

[1565] System configuration

[1566] This system consists of the following main components:

[1567] 1. Data Collection Module

[1568] 2. Data Analysis Module

[1569] 3. Anomaly Detection Module

[1570] 4. Warning Module

[1571] 5. User preference recording module

[1572] 6. Voice Dialogue Module

[1573] 7. Emotion Engine Module

[1574] 8. Remote monitoring interface

[1575] Collection and analysis of electricity, water and gas usage information

[1576] The server collects real-time electricity, water, and gas usage information from smart meters installed in each home or building. This allows the data collection module to centrally manage various usage information. The server analyzes the collected data and compares it with past data to detect anomalies.

[1577] Webcam footage collection and analysis

[1578] The server captures real-time video footage from inside the home via a webcam, which is then analyzed using machine learning technology to detect abnormal behavior such as falls.

[1579] Emotion recognition by emotion engine

[1580] The server captures the user's audio and video data and analyzes it with an emotion engine module, which uses machine learning models to analyze the user's facial expressions, tone of voice, and other factors to recognize the user's emotions in real time.

[1581] Response after detecting an anomaly

[1582] The server issues a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. If a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[1583] Emotion-based responses

[1584] The server generates appropriate voice dialogue based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, it can recommend relaxing music. It can also automatically adjust the brightness of lights and music according to the user's emotions.

[1585] Providing features based on user preferences

[1586] The server analyzes the user's past viewing history and preferences and automatically records programs that the user may be interested in. The recording reservation list is updated in real time, providing the user with an optimized entertainment experience.

[1587] Voice interaction with the user

[1588] The server uses a generative AI model to communicate with the user through voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated. For example, it is possible to have a conversation about everyday information such as the day's schedule or the weather.

[1589] Remote Monitoring and Operation

[1590] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to take appropriate action quickly when an abnormality occurs or when regular checks are required.

[1591] Specific examples

[1592] Example 1: Program recommendation using emotion recognition

[1593] If the server recognizes through its emotion engine that the user is feeling stressed, it will recommend relaxing movies and TV shows that suit the user's preferences and automatically record them.

[1594] Example 2: Adjusting the environment based on emotions

[1595] If the server recognizes through its emotion engine that the user is feeling tired, it will dim the lights in the room and play relaxing music.

[1596] Example 3: Emergency response with fall detection and emotion recognition

[1597] If the server detects a fall and the emotion engine recognizes pain or confusion from the user's facial expression, it will immediately activate the emergency call system and dispatch an ambulance.

[1598] In this way, the present invention makes full use of advanced technology to achieve safety, comfort, and optimal response according to the user's emotions.

[1599] The processing flow will be explained below.

[1600] Step 1:

[1601] The server obtains electricity usage information from the smart meter every minute via an API.

[1602] Step 2:

[1603] The server obtains water usage information from the water meter every 10 minutes via an API.

[1604] Step 3:

[1605] The server obtains gas usage information from the gas meter every 15 minutes via an API.

[1606] Step 4:

[1607] The server receives and temporarily stores the video stream from the webcam in real time.

[1608] Step 5:

[1609] The server analyzes the collected power usage information and compares it with data from the past 24 hours to detect abnormally high power usage.

[1610] Step 6:

[1611] The server analyzes water usage information and detects abnormalities when water is used continuously for long periods of time.

[1612] Step 7:

[1613] The server analyzes gas usage information and monitors sudden fluctuations in gas usage in real time.

[1614] Step 8:

[1615] The server analyzes webcam footage using machine learning models to detect abnormal behavior such as the user falling.

[1616] Step 9:

[1617] The server captures the user's voice in real time and analyzes it with an emotion engine to recognize emotions.

[1618] Step 10:

[1619] The server acquires the user's facial expression data from a webcam and analyzes it with an emotion engine to recognize emotions.

[1620] Step 11:

[1621] The server generates appropriate voice dialogue for the user based on the recognized emotion. For example, if the user looks sad, the server might say, "Is there something I can help you with?"

[1622] Step 12:

[1623] Based on the recognized emotions, the server recommends TV programs and movies that match the user's preferences and automatically records them.

[1624] Step 13:

[1625] The server will send a warning alert to the user if an abnormality is detected. For example, if water continues to flow for a long period of time, it will immediately notify the user that there may be a water leak.

[1626] Step 14:

[1627] The server will activate the emergency call system and dispatch an ambulance if a fall is detected, and will also initiate emergency response if the emotion engine detects emotions such as pain or confusion.

[1628] Step 15:

[1629] If the server detects an emotion requiring relaxation through its emotion engine, it will dim the lights slightly and play relaxing music.

[1630] Step 16:

[1631] The server detects that the user has returned home via the entrance sensor and plays a voice message saying "Welcome home."

[1632] Step 17:

[1633] Administrators and family members can access a dedicated remote monitoring interface from the device and check various data in real time.

[1634] Step 18:

[1635] Remotely control electricity, water, and gas usage as needed from your device.

[1636] By performing detailed processing step by step in this way, the safety and comfort of the entire system and optimal responses according to the user's emotions are ensured.

[1637] Example 2

[1638] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1639] Conventional monitoring systems collect electricity, water, and gas usage information and webcam footage, but are limited in their ability to comprehensively analyze this information and detect abnormalities. They also lack the means to analyze a user's emotional state in real time and take appropriate action based on the abnormality or emotion. Furthermore, they lack the functionality to recommend content based on the user's preferences and generate voice dialogue. The present invention aims to solve these problems and provide a safe and comfortable living environment.

[1640] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1641] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing an alert when an abnormality is detected, means for recording programs according to a user's preferences, means for engaging in voice conversation with the user, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, means for analyzing the user's emotions, means for adjusting the environment based on the user's emotions, and means for generating voice dialogue using a generative AI model. This enables the server to comprehensively analyze electricity, water, and gas usage information and video data, detect abnormalities, and take appropriate measures, as well as adjust the environment, recommend content, and engage in voice dialogue according to the user's emotional state.

[1642] "Electricity usage information" refers to data on the usage of electricity within homes and buildings.

[1643] "Water usage information" refers to data such as the amount of water used within a home or building and the timing of use.

[1644] "Gas usage information" refers to data regarding the amount and timing of gas usage within homes and buildings.

[1645] "Webcam Footage" refers to video footage data collected through a webcam.

[1646] "Means for detecting anomalies" refers to algorithms or analytical systems that identify anomalies by comparing them with typical usage and historical data.

[1647] "Means of issuing an alert" refers to a system or function that sends a notification to a user or administrator when an abnormality is detected.

[1648] "Means for recording programs according to the user's tastes and preferences" refers to a system that automatically records highly relevant programs based on the user's past viewing history and preferences.

[1649] "Means of interacting with the user via voice" refers to a system that uses speech recognition and generative AI models to provide appropriate responses to user utterances.

[1650] "Means for detecting the user's return home using a front door sensor and playing a voice message" refers to a system that detects the user's return home based on information obtained from a front door sensor and plays a pre-set voice message.

[1651] "Means for providing an interface for remote monitoring and operation" refers to an interface that allows building managers and family members to monitor the situation within the home in real time via the Internet and perform necessary operations.

[1652] "Means for analyzing user emotions" refers to a system that analyzes the user's facial expressions and voice data and recognizes their emotional state in real time.

[1653] "Means for adjusting the environment based on the user's emotions" refers to a system that automatically adjusts environmental settings such as room lighting and music according to the user's emotional state.

[1654] "Means for generating a voice dialogue using a generative AI model" refers to a system that uses an AI model to generate a voice dialogue with a user in real time.

[1655] This system combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect anomalies, with an emotion engine that recognizes user emotions. This system consists of the following main components:

[1656] System configuration

[1657] 1. Data Collection Module

[1658] 2. Data Analysis Module

[1659] 3. Anomaly Detection Module

[1660] 4. Warning Module

[1661] 5. User preference recording module

[1662] 6. Voice Dialogue Module

[1663] 7. Emotion Engine Module

[1664] 8. Remote monitoring interface

[1665] Collection and analysis of electricity, water and gas usage information

[1666] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building and manages it centrally. This allows the data collection module to periodically update the usage information. The collected data is stored in a database and processed by the data analysis module. For example, an algorithm is applied to compare the data with past data to detect anomalies in usage.

[1667] Webcam footage collection and analysis

[1668] The server captures real-time video footage from inside the home via a webcam. This video data is analyzed using machine learning technology to detect abnormal behavior, such as falls. If an abnormality is detected, the anomaly detection module immediately issues an alert and, if necessary, activates the emergency call system.

[1669] Emotion recognition by emotion engine

[1670] The server acquires the user's audio and video data and analyzes it with an emotion engine module. This emotion engine uses machine learning models to analyze the user's facial expressions, tone of voice, and other factors to recognize their emotional state in real time.

[1671] Response after detecting an anomaly

[1672] The server has a means to issue a warning if it detects an abnormality. For example, if the water supply is used continuously for a long period of time, it may indicate a water leak and immediately notifies the user. Also, if a fall is detected, the emergency notification system will automatically activate and dispatch an ambulance.

[1673] Emotion-based responses

[1674] The server generates appropriate voice dialogue based on the user's emotions recognized by the emotion engine. If the user is feeling stressed, it can recommend relaxing music and adjust the lighting in the room. Other features include recommending entertainment content based on the user's emotional state.

[1675] Providing features based on user preferences

[1676] The server analyzes the user's past viewing history and preferences, and has the function of recommending and recording content that the user may be interested in. The recording reservation list is updated in real time to optimize the entertainment experience.

[1677] Voice interaction with the user

[1678] The server uses a generative AI model to converse with the user via voice. When the user speaks into the microphone, the content is analyzed and an appropriate response is generated in real time. For example, it is possible to have a conversation about everyday information such as the day's schedule, weather, and news.

[1679] Remote Monitoring and Operation

[1680] Building managers and family members can access a dedicated remote monitoring interface from a terminal and monitor and control the situation inside the home in real time, allowing them to respond quickly and appropriately when an abnormality occurs or regular checks are required.

[1681] Examples and prompts

[1682] Example 1: Program recommendation using emotion recognition

[1683] If the server recognizes through its emotion engine that the user is feeling stressed, it will recommend relaxing movies and TV shows that suit the user's preferences and automatically record them.

[1684] Prompt: "Explain how an emotion recognition engine can recommend relaxing content when it detects stress."

[1685] Example 2: Adjusting the environment based on emotions

[1686] When the server's emotion engine recognizes that the user is feeling tired, it dims the lights in the room and plays relaxing music.

[1687] Prompt: "Describe how you can automatically adjust the environment in a room based on your emotional state."

[1688] Example 3: Emergency response with fall detection and emotion recognition

[1689] If the server detects a fall and the emotion engine recognizes pain or confusion from the user's facial expression, the emergency call system will immediately be activated and an ambulance will be dispatched.

[1690] Prompt: "Please describe your emergency response process when you detect a client has fallen."

[1691] In this way, the present invention utilizes advanced technology to provide optimal responses based on the user's safety, comfort, and emotions.

[1692] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1693] Step 1: Data collection

[1694] The server collects electricity, water, and gas usage information in real time from smart meters installed in each home or building. This data is input into a data collection module and stored centrally in a database. Specifically, the data sent from the smart meters is obtained using the HTTP protocol and stored as is in the database.

[1695] Input: Real-time data from smart meters

[1696] Output: Electricity, water, and gas usage information stored in a database

[1697] Step 2: Data analysis

[1698] The server analyzes the electricity, water, and gas usage information stored in the database. This involves comparing usage with past data and using algorithms to detect patterns. Specifically, it compares data from the past month and performs statistical analysis to identify sudden increases or decreases in usage.

[1699] Input: Historical electricity, water, and gas usage information stored in a database

[1700] Output: Statistical data for anomaly detection

[1701] Step 3: Anomaly detection

[1702] The server uses the statistical data obtained from the data analysis module to detect anomalies. If an anomaly is detected, the information is input into the anomaly detection module, which generates a warning message. Specifically, if an abnormal pattern is discovered, an alert flag is set and sent to the user notification system.

[1703] Input: Statistical analysis data

[1704] Output: Anomaly detection alert flag and warning message

[1705] Step 4: Collect and analyze webcam footage

[1706] The server collects real-time video data from the webcam and analyzes it using machine learning technology. Specifically, the video analysis algorithm detects abnormal behavior such as falls and issues an alert if necessary.

[1707] Input: Real-time video data from a webcam

[1708] Output: Detected abnormal behavior and warning message

[1709] Step 5: Emotion Recognition

[1710] The server inputs the user's audio and video data into the emotion engine module and analyzes their emotional state. This allows the system to recognize emotions in real time based on the user's facial expressions and tone of voice. Specifically, the emotion engine analyzes the user's audio and video data and generates emotion tags.

[1711] Input: Audio and video data

[1712] Output: Emotion tag

[1713] Step 6: Dealing with abnormalities and emotions

[1714] The server responds appropriately based on the abnormality and emotional state. If an abnormality occurs, the emergency notification system is activated, and depending on the emotional state, the response may include adjusting the music or lighting. Specifically, if an abnormality is detected, a warning message is sent, and if the emotional state is identified, a music playback API is called to play a playlist.

[1715] Input: Anomaly detection alert flag or sentiment tag

[1716] Output: Emergency call, music playback, lighting control

[1717] Step 7: User Preference Management

[1718] The server analyzes the user's past viewing history and preference data, and recommends and records content that the user may be interested in. Specifically, it queries the viewing history database, and the recommendation engine suggests appropriate content.

[1719] Input: Viewing history database

[1720] Output: Recommended content

[1721] Step 8: Voice interaction generation

[1722] The server uses a voice interaction module to communicate with the user via voice. The generative AI model analyzes what the user says and generates an appropriate response. Specifically, the server converts the user's voice input into text, inputs that text into the generative AI model to generate a response, and then converts that response into voice.

[1723] Input: User voice input

[1724] Output: Voice reply

[1725] Step 9: Remote monitoring and operation

[1726] The terminal allows building managers and family members to access the remote monitoring interface and monitor and control the situation inside the home in real time.Specifically, they log in to the remote monitoring interface, check camera footage and smart meter data inside the home, and remotely change lighting and air conditioning settings as needed.

[1727] Input: Real-time monitoring data

[1728] Output: Operation instructions

[1729] (Application example 2)

[1730] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1731] Conventional anomaly detection systems monitor usage information for electricity, water, gas, etc. in real time, and have the ability to detect abnormalities and issue alarms, but they are unable to grasp the emotional state of the user and take appropriate action.In addition, there are limitations to the means of quickly notifying households and administrators when an abnormality occurs, so further improvements in safety and comfort are needed.

[1732] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1733] In this invention, the server includes means for collecting electricity usage information in real time, means for collecting water usage information in real time, means for collecting gas usage information in real time, means for collecting webcam footage in real time, means for analyzing the electricity usage information, water usage information, gas usage information, and webcam footage to detect abnormalities, means for issuing a warning when an abnormality is detected, means for recording programs according to a user's preferences, means for audio communication with the user, means for detecting the user's return home using a front door sensor and playing a voice message, means for providing an interface for remote monitoring and operation, means for recognizing a user's emotion, means for providing warnings and advice based on the user's emotional state, means for detecting abnormalities based on the electricity, water, and gas usage information and sending an electronic message, and means for making an emergency call based on the webcam footage. This enables not only a rapid response when an abnormality is detected but also the provision of advanced warnings and advice based on the user's emotional state.

[1734] - "Electricity usage information" means data indicating the amount of electricity consumed within a home or facility.

[1735] "Water usage information" is data that indicates the amount of water used within a home or facility.

[1736] "Gas usage information" means data indicating the amount of gas used within a home or facility.

[1737] "Webcam footage" refers to real-time video data captured using a webcam.

[1738] "Means for detecting anomalies" refers to analytical means for identifying unusual conditions or behaviors from collected data.

[1739] "Means for issuing warnings" refers to notification means for alerting users and administrators when an abnormality is detected.

[1740] The "means for recording programs according to the user's tastes and preferences" is a means for automatically recording programs and video content that the user likes.

[1741] "Means for communicating with the user by voice" refers to an interface that responds to and communicates with the user by voice.

[1742] The "entrance sensor" is a sensor that detects whether the entrance is open or closed.

[1743] The "means for playing back a voice message" is a means for playing back a voice message when the user returns home.

[1744] "Interface for remote monitoring and operation" refers to an interface that allows a manager or family member in a remote location to monitor and operate the situation within the home in real time.

[1745] "Means for recognizing user emotions" refers to means for identifying the user's emotional state by analyzing the user's facial expressions and voice.

[1746] The "means for providing warnings and advice based on emotional state" refers to means for providing appropriate warnings and advice to a user in accordance with the recognized emotional state.

[1747] "Means for sending electronic messages" refers to means for sending electronic messages to users or administrators when an abnormality is detected.

[1748] "Means for making an emergency call" refers to a means for making an emergency call when an abnormality that poses a threat to human life is detected based on webcam footage.

[1749] As a form for implementing the present invention, we will explain a specific embodiment of a security service that combines a system that collects and analyzes electricity usage information, water usage information, gas usage information, and webcam footage in real time to detect abnormalities, with an emotion engine that recognizes user emotions.

[1750] System configuration

[1751] This system consists of the following main components:

[1752] 1. Data Collection Module

[1753] The server uses smart meters and web cameras to obtain electricity usage information, water usage information, gas usage information, and real-time images from inside the home.

[1754] 2. Data Analysis Module

[1755] The server analyzes the collected electricity, water, and gas usage information and webcam footage to detect abnormalities, and identifies them by comparing them with past usage data.

[1756] 3. Anomaly Detection Module

[1757] The server will issue a warning to the user if it detects any abnormalities from the analysis results, such as if the water supply has been running for an extended period of time or if it detects abnormal behavior based on webcam footage.

[1758] 4. Emotion Engine Module

[1759] The server recognizes the user's emotions using webcam footage and audio data acquired from a microphone. This emotion engine uses a machine learning model to analyze facial expressions and tone of voice to grasp the user's emotional state in real time.

[1760] 5. Warning Module

[1761] When an abnormality is detected, the server will send a warning to the user via email or push notification. In addition, if the emotion engine recognizes that the user is feeling stressed or anxious, it will provide the user with appropriate advice.

[1762] 6. Voice Dialogue Module

[1763] The server uses a generative AI model to interact with the user through voice: when the user speaks into the microphone, the content is analyzed and an appropriate response is generated.

[1764] 7. Emotion-based responses

[1765] The server can recommend relaxing music or adjust the lighting in the room based on the user's emotions recognized by the emotion engine.

[1766] As a specific example, if a user is feeling stressed, the system may notify them, "You seem to be feeling stressed. I'll play some music to help you relax," and play a relaxing song from the user's music library.

[1767] Examples of prompts to be input to a generative AI model include, "The user's emotions have changed. Please provide a response based on their latest emotional state," and "An anomaly has been detected. Please suggest an appropriate response."

[1768] 8. Remote monitoring interface

[1769] Administrators and family members can access a dedicated remote monitoring interface from a terminal and monitor and operate the system in real time, allowing for immediate response when an abnormality occurs or when regular checks are required.

[1770] This system uses hardware such as smartphones, webcams, and smart meters, and software such as Python, OpenCV, and Keras (with TensorFlow backend). The server collects, analyzes, and processes data from these hardware devices to improve user safety and comfort.

[1771] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1772] Step 1:

[1773] Data collection

[1774] The server collects electricity usage information, water usage information, gas usage information, and webcam footage in real time from smart meters and webcams. The input is data from each sensor and device, and the output is real-time data that is stored in a database on the server.

[1775] Step 2:

[1776] Data analysis

[1777] The server compares collected electricity, water, and gas usage information with past data. It also performs face detection and abnormal behavior detection on webcam footage. The input is real-time and past data, and the output is an alert indicating whether or not an abnormality has occurred.

[1778] Step 3:

[1779] Anomaly detection

[1780] The server detects anomalies from the analysis results. If an anomaly is detected, it generates a warning message. The input is the analysis results, and the output is the warning message.

[1781] Step 4:

[1782] emotion recognition

[1783] The server uses the webcam video and microphone audio data to recognize the user's emotions with an emotion engine. The input is real-time video and audio data, and the output is data representing the user's emotional state.

[1784] Step 5:

[1785] Warning

[1786] If an anomaly is detected or if the emotion engine recognizes the user's stress or anxiety, the server issues a warning to the user via email, push notification, etc. The input is the anomaly detection result and emotion recognition result, and the output is email or push notification.

[1787] Step 6:

[1788] Voice dialogue

[1789] The server analyzes what the user says into the microphone using a generative AI model and generates an appropriate response. The input is voice data, and the output is the generated response.

[1790] Step 7:

[1791] Emotion-based responses

[1792] The server recommends music suitable for the user and adjusts lighting based on the recognition results of the emotion engine. The input is emotion recognition data, and the output is commands to play music or control lighting.

[1793] Step 8:

[1794] Remote Monitoring

[1795] Administrators and family members can access a dedicated remote monitoring interface from a terminal to monitor and operate the situation in real time. The input is monitoring data and operation instructions, and the output is the monitoring results and the executed operations.

[1796] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1797] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1798] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1799] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1800] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1801] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1802] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1803] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1804] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1805] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1806] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1807] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1808] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1809] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1810] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1811] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1812] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1813] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1814] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1815] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1816] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1817] The following is further disclosed regarding the above embodiment.

[1818] (Claim 1)

[1819] a means for collecting electricity usage information in real time;

[1820] a means of collecting water usage information in real time;

[1821] a means for collecting gas usage information in real time;

[1822] a means for collecting webcam footage in real time;

[1823] means for analyzing the electricity usage information, the water usage information, the gas usage information, and the web camera image to detect an abnormality;

[1824] means for issuing an alarm when an abnormality is detected;

[1825] means for recording programs according to the user's tastes and preferences;

[1826] means for audibly interacting with a user;

[1827] a means for detecting the user's return home by a door sensor and playing a voice message;

[1828] A system including a means for providing an interface for remote monitoring and operation.

[1829] (Claim 2)

[1830] The system according to claim 1, further comprising means for automatically calling an ambulance when the abnormality is detected.

[1831] (Claim 3)

[1832] The system of claim 1 , further comprising means for sending a notification to a building manager or a family member via a remote monitoring interface when the abnormality is detected.

[1833] "Example 1"

[1834] (Claim 1)

[1835] a means for collecting electricity usage information in real time;

[1836] a means of collecting water usage information in real time;

[1837] a means for collecting gas usage information in real time;

[1838] a means for collecting webcam footage in real time;

[1839] means for analyzing the electricity usage information, the water usage information, the gas usage information, and the web camera image using a data analysis module to detect an abnormality;

[1840] means for issuing an alarm when an abnormality is detected;

[1841] means for recording programs using a user preference analysis module;

[1842] A means of interacting with the user via voice using a generative AI model;

[1843] A system including means for monitoring and operating through a remote monitoring interface.

[1844] (Claim 2)

[1845] The system according to claim 1, further comprising means for automatically calling an ambulance when the abnormality is detected.

[1846] (Claim 3)

[1847] The system of claim 1 , further comprising means for sending a notification to a manager or family member via a remote monitoring interface when the abnormality is detected.

[1848] "Application Example 1"

[1849] (Claim 1)

[1850] a means for collecting electricity usage information in real time;

[1851] a means of collecting water usage information in real time;

[1852] a means for collecting gas usage information in real time;

[1853] a means for collecting webcam footage in real time;

[1854] means for analyzing the electricity usage information, the water usage information, the gas usage information, and the web camera image to detect an abnormality;

[1855] means for issuing an alarm when an abnormality is detected;

[1856] means for recording programs according to the user's tastes and preferences;

[1857] means for audibly interacting with a user;

[1858] a means for detecting the user's return home by a door sensor and playing a voice message;

[1859] means for providing an interface for remote monitoring and operation;

[1860] A means for detecting movement by a webcam and notifying an abnormality when movement is detected;

[1861] A system including:

[1862] (Claim 2)

[1863] 10. The system of claim 1, further comprising means for automatically calling an ambulance when an abnormality is detected.

[1864] (Claim 3)

[1865] 10. The system of claim 1, further comprising means for sending a notification to a building manager or a family member via the remote monitoring interface when an abnormality is detected.

[1866] "Example 2: Combining Emotion Engines"

[1867] (Claim 1)

[1868] a means for collecting electricity usage information in real time;

[1869] a means of collecting water usage information in real time;

[1870] a means for collecting gas usage information in real time;

[1871] a means for collecting webcam footage in real time;

[1872] means for analyzing the electricity usage information, the water usage information, the gas usage information, and the web camera image to detect an abnormality;

[1873] means for issuing an alarm when an abnormality is detected;

[1874] means for recording programs according to the user's tastes and preferences;

[1875] means for audibly interacting with a user;

[1876] a means for detecting the user's return home by a door sensor and playing a voice message;

[1877] means for providing an interface for remote monitoring and operation;

[1878] means for analyzing user emotions;

[1879] means for adjusting the environment based on the user's emotions;

[1880] A system including means for generating a voice interaction using a generative AI model.

[1881] (Claim 2)

[1882] The system according to claim 1, further comprising means for automatically calling an ambulance when the abnormality is detected.

[1883] (Claim 3)

[1884] The system of claim 1 , further comprising means for sending a notification to a building manager or a family member via a remote monitoring interface when the abnormality is detected.

[1885] "Application example 2 when combining emotion engines"

[1886] (Claim 1)

[1887] a means for collecting electricity usage information in real time;

[1888] a means of collecting water usage information in real time;

[1889] a means for collecting gas usage information in real time;

[1890] a means for collecting webcam footage in real time;

[1891] means for analyzing the electricity usage information, the water usage information, the gas usage information, and the web camera image to detect an abnormality;

[1892] means for issuing an alarm when an abnormality is detected;

[1893] means for recording programs according to the user's tastes and preferences;

[1894] means for audibly interacting with a user;

[1895] a means for detecting the user's return home by a door sensor and playing a voice message;

[1896] means for providing an interface for remote monitoring and operation;

[1897] means for recognizing a user's emotion;

[1898] means for providing warnings and advice based on the user's emotional state;

[1899] A means for detecting anomalies based on electricity, water, and gas usage information and sending an electronic message;

[1900] A means of making emergency calls based on webcam footage;

[1901] A system including:

[1902] (Claim 2)

[1903] The system according to claim 1, further comprising means for automatically calling an ambulance when the abnormality is detected.

[1904] (Claim 3)

[1905] The system of claim 1 , further comprising means for sending a notification to a manager or family member via a remote monitoring interface when the abnormality is detected. [Explanation of symbols]

[1906] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smar...

Claims

1. a means for collecting electricity usage information in real time; a means of collecting water usage information in real time; a means for collecting gas usage information in real time; a means for collecting webcam footage in real time; means for analyzing the electricity usage information, the water usage information, the gas usage information, and the web camera image to detect an abnormality; means for issuing an alarm when an abnormality is detected; means for recording programs according to the user's tastes and preferences; means for audibly interacting with a user; a means for detecting the user's return home by a door sensor and playing a voice message; A system including a means for providing an interface for remote monitoring and operation.

2. The system according to claim 1 , further comprising means for automatically calling an ambulance when the abnormality is detected.

3. The system of claim 1 , further comprising means for sending a notification to a building manager or a family member via a remote monitoring interface when the abnormality is detected.

Images