system

Abstract

We provide the system. [Solution] A means for obtaining the usage status of electricity, water, and gas within a household via one or more sensors, The acquired data includes means for noise reduction and missing value processing, A means of learning user lifestyle patterns by analyzing pre-processed data using an artificial intelligence model, A means for setting rules or thresholds to detect abnormal patterns based on the analyzed data, A means of sending an alarm to a pre-configured recipient when an abnormal pattern is detected, A means for adjusting the parameters of the artificial intelligence model based on user feedback, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, it is important for family members living in distant places to regularly check the safety and health status of the elderly. However, due to limited means, there is a current situation where it is difficult for the elderly to live with peace of mind. Many existing monitoring systems require the elderly themselves to operate specific devices or applications, which are cumbersome to operate for the elderly and may cause resistance to technology, resulting in cases where they are not fully utilized. In addition, there is also a problem that changes and abnormalities in the daily rhythm cannot be detected in real time, so it is impossible to respond promptly even when an abnormality occurs.

Means for Solving the Problems

[0005] To solve these problems, the present invention provides a means for acquiring household electricity, water, and gas usage via one or more sensors. Furthermore, it includes means for preprocessing the acquired data to remove noise and handle missing values. By analyzing the preprocessed data using an artificial intelligence model and learning the user's lifestyle patterns, it provides a means for detecting abnormalities in daily routines. It also includes means for sending an alarm to a pre-set recipient when an abnormality is detected. Furthermore, by utilizing user feedback and adjusting the parameters of the artificial intelligence model, the accuracy of detection can be improved. This enables elderly people to naturally monitor their daily lives without performing specific operations and to immediately notify family members or caregivers of any abnormalities, thus creating a system that can do so.

[0006] A "sensor" is a device that detects a specific physical phenomenon and outputs that information as an electrical signal.

[0007] "Electricity usage" refers to information that shows the amount and pattern of electrical energy consumed within a household.

[0008] "Water usage data" refers to information indicating the amount of water used within a household and the timing of its use.

[0009] "Gas usage" refers to information that shows the amount of gas energy consumed in a household and the time of day it is used.

[0010] "Noise reduction" is the process of removing unnecessary data and errors in order to accurately obtain the necessary information when analyzing data.

[0011] "Missing value handling" is the process of filling in the missing information in the acquired data.

[0012] An "artificial intelligence model" is a set of mathematical methods and algorithms that learn through data analysis and discover specific patterns or rules.

[0013] A "lifestyle pattern" refers to a specific sequence of actions and habits in an individual's daily life that are repeated regularly.

[0014] An "abnormal pattern" is an indicator that shows behavior or conditions that deviate from normal lifestyle patterns.

[0015] "Sending an alarm" means notifying designated recipients of the details of an anomaly that has been detected.

[0016] "Feedback" refers to information obtained from user reactions and evaluations that is used to adjust and improve the functions and performance of a system. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0021] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] ​In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] This invention is a system for monitoring the living conditions of elderly people. It combines machine learning technology to analyze the usage of electricity, water, and gas within the home, monitors living patterns, and detects abnormalities. The operation flow of this system is described below.

[0039] First, several types of sensors are installed in the user's home. These sensors include power consumption meters, flow sensors, and voice sensors. These sensors monitor the use of electricity, gas, and water in the home in real time.

[0040] Data obtained from each sensor is transmitted to a server via the network through a terminal. The server removes noise from the received data and appropriately imputes missing data. This preprocessing is performed to improve the accuracy of the analysis.

[0041] Next, the server inputs the pre-processed data into an artificial intelligence model for analysis. This AI model learns the user's lifestyle patterns based on past data and builds a model of normal behavior. For example, it can detect when electricity consumption increases each morning and recognize the user's wake-up pattern.

[0042] Based on established usage patterns, the server sets criteria for detecting anomalies. These include deviations from normal usage patterns and periods of absence from use. When an anomaly is detected, the server sends an alert to the user's family or caregiver's device.

[0043] As a concrete example, suppose an elderly person has a routine of using an electric kettle at 8 AM every morning, followed by a shower. If this routine suddenly stops completely, the server will determine it's an anomaly based on defined criteria and send an alert. The user's family can receive a notification on their smartphone and quickly check the situation.

[0044] Finally, based on user feedback, the server continuously refines its machine learning model to further improve the system's accuracy. This approach enables continuous monitoring without requiring elderly individuals to operate specific technological devices.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] Various sensors installed in the user's home measure the usage of electricity, water, gas, and voice, and transmit the data to a terminal.

[0048] Step 2:

[0049] The terminal converts the collected data into a format suitable for transfer to the server and sends it to the server via the network.

[0050] Step 3:

[0051] The server performs noise reduction processing and handles missing values ​​on the received raw data. Specifically, it cleans up the data using appropriate filtering techniques.

[0052] Step 4:

[0053] The server inputs pre-processed data into an artificial intelligence (AI) analysis unit, uses a trained model to analyze the user's daily life patterns, and constructs a baseline daily rhythm.

[0054] Step 5:

[0055] The server sets anomaly detection criteria based on the established lifestyle rhythm, and monitors for abnormal patterns while performing real-time data analysis.

[0056] Step 6:

[0057] When an anomaly is detected, for example, if there is no usage for an extended period during normal operating hours, the server generates an alert and prepares to send a warning to pre-registered recipients.

[0058] Step 7:

[0059] The device (for example, the user's family member or caregiver's smartphone) receives an alert notification from the server and immediately displays the notification to the user. This allows them to check the situation.

[0060] Step 8:

[0061] If a false positive occurs based on user feedback, the server receives the feedback data, adjusts the AI ​​model, and improves the accuracy of subsequent analyses.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] There is a need to provide a safe and secure living environment for the elderly and those requiring care, but conventional monitoring systems lack the accuracy to detect abnormal behavior in real time. Furthermore, it is difficult to conduct unbiased monitoring while protecting user privacy. It is necessary to solve these problems and realize a safe and efficient monitoring system.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for acquiring the usage status of energy resources in the environment via one or more sensing devices, means for performing noise reduction and missing information imputation for preprocessing the acquired information, and means for analyzing the preprocessed information using a machine learning algorithm to learn the behavioral patterns of the person being monitored. This enables high-precision monitoring of the person's lifestyle and allows for a quick and appropriate response when an anomaly is detected. Furthermore, it enables continuous monitoring while protecting privacy.

[0067] A "sensing device" is a device used to measure the usage of energy resources such as electricity, water, and gas in the environment and to acquire that information.

[0068] "Information preprocessing" is a process that improves data quality by removing noise and filling in missing information from acquired data.

[0069] A "machine learning algorithm" is a computational method that identifies patterns from large amounts of data and learns from them to make predictions and classifications about future data.

[0070] A "behavioral pattern" is a set of characteristics that indicate the regularity or tendencies of the daily activities and behaviors of the person being monitored.

[0071] An "abnormal pattern" refers to irregular activity or behavior that differs from the normal behavioral patterns learned by a machine learning algorithm.

[0072] A "criterion or threshold" is a criterion or limit used to make a judgment when detecting an unusual pattern.

[0073] A "warning signal" is an alarm message sent to a pre-configured recipient when an unusual pattern is detected.

[0074] A "mobile communication device" is a portable device that a user can carry and use to receive communications and notifications from external sources.

[0075] This invention is a monitoring system primarily for safely observing the lives of the elderly and those requiring care. The system operates as follows:

[0076] Multiple sensing devices are installed in the user's environment. These devices measure the usage of energy resources such as electricity, water, and gas in real time. This allows for detailed monitoring of usage patterns in daily life.

[0077] The collected information is transmitted to the server via the terminal. Upon arrival at the server, the information undergoes preprocessing, including noise reduction and the imputation of missing information. This preprocessing improves the accuracy of the data, enabling more reliable analysis.

[0078] The pre-processed data is then analyzed using a machine learning algorithm. The server uses this algorithm to learn the behavioral patterns of the monitored individuals. For example, it can identify patterns such as whether the monitored individual uses an electric kettle at a specific time each day.

[0079] Based on the analysis results, the server sets criteria and thresholds for detecting abnormal behavior. This process allows for rapid detection of unusual behavior.

[0080] When an anomaly is detected, the server sends an alert signal to pre-configured recipients. For example, if a monitored person stops using an electrical appliance they normally use, an alert can be quickly sent to their family. This alert signal is delivered as a notification to the user's mobile device, allowing for prompt action.

[0081] Furthermore, by utilizing feedback from users and their families, the server continuously adjusts the parameters of its machine learning algorithms to improve the system's accuracy. This dynamic adjustment allows the system to flexibly adapt to changing environments and new behavioral patterns.

[0082] As described above, this system uses a generative AI model to achieve effective monitoring. For example, by inputting prompts such as, "How can we improve the way we learn the lifestyle patterns of elderly people and notify them of abnormalities?" into the generative AI model, it is possible to find even more efficient solutions.

[0083] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0084] Step 1:

[0085] Users install multiple sensing devices in their homes. These devices collect real-time data on electricity, water, and gas usage. The input is data on the use of various energy resources, and the output is time-series usage patterns of the surrounding environment. This process enables detailed monitoring of the residents' lives.

[0086] Step 2:

[0087] The terminal transmits the collected data to the server via the network. The input is raw data from the sensing device, and the output is secure data packets that reach the server. Here, data encryption technology is used to ensure the security of the information and prevent unauthorized access from external sources.

[0088] Step 3:

[0089] The server preprocesses the received data. Specifically, it applies a denoising filter and imputes missing data based on statistical methods. The input is raw data sent from the terminal, and the output is a reliable and clean dataset. This process improves analytical performance.

[0090] Step 4:

[0091] The server inputs pre-processed data into a machine learning model and performs the analysis. The algorithm repeatedly learns from the data and extracts the user's typical behavioral patterns. The input is a clean dataset, and the output generates the behavioral patterns of the monitored individual. This analysis identifies daily habits.

[0092] Step 5:

[0093] The server sets anomaly detection criteria based on the analysis results. A threshold is set, and any behavior exceeding it is detected as an anomaly. The input is behavioral pattern data, and the output is the formulation of anomaly detection rules. This enables the immediate detection of behavior outside the normal range.

[0094] Step 6:

[0095] If an anomaly is detected, the server sends a warning signal. An alarm is sent to the terminal of a pre-configured recipient. The input is the result of the anomaly detection algorithm, and the output is an alert notification sent to the recipient. This notification prompts a quick response.

[0096] Step 7:

[0097] The user or their representative sends feedback to the server. The server uses this feedback to adjust the parameters of the machine learning model, improving the system's accuracy. The input is the feedback information, and the output is the optimized model parameters. This adjustment leads to continuous improvement.

[0098] (Application Example 1)

[0099] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] There is a need to efficiently and continuously detect abnormalities within the home and respond quickly while ensuring the safety of the elderly. Conventional systems only monitor electricity and water usage, making it difficult to immediately detect unauthorized access from external sources or suspicious usage patterns. This invention aims to solve these problems and enhance the safety of the home and its surrounding environment.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes means for acquiring energy and liquid usage within the home via one or more detection means; means for noise reduction and missing value processing for preprocessing the acquired information; means for analyzing the preprocessed information using a machine learning model to learn user behavior patterns; and means for detecting unauthorized use from external sources, recording the anomaly, and notifying the user. This enables early detection of potential dangers faced by the elderly within the home and allows for a rapid response.

[0103] "Detection means" refers to a device or group of devices installed to monitor energy and liquid usage within a household.

[0104] "Preprocessing" is the process of removing noise from acquired information and filling in missing values.

[0105] A "machine learning model" is an artificial intelligence-based algorithm used to learn and analyze user behavior patterns from pre-processed information.

[0106] A "rule or criterion" is a rule or threshold set to detect anomaly patterns based on the analyzed information.

[0107] "Communication means" refers to a means of sending an alarm to a pre-configured recipient when an abnormal pattern is detected.

[0108] "Evaluation" refers to opinions and information regarding the operation of a system received as feedback from users.

[0109] "Conditions" refer to the parameters or settings of a machine learning model that are changed when tuning the model.

[0110] "Detecting unauthorized use" refers to the process of discovering unauthorized use or access from outside the home or its surroundings.

[0111] "Energy and liquids" refers to infrastructure resources such as electricity and water consumed within the home.

[0112] The system for implementing this invention has the function of monitoring and analyzing household electricity and water usage to detect anomalies in order to ensure the safety of the elderly. This system is operated using a server, terminals, a group of sensors, and a machine learning model.

[0113] The server receives data acquired from multiple sensors installed within the home, including power consumption meters and flow sensors. The received data undergoes noise reduction and interpolation on the server, and is pre-processed for analysis.

[0114] Next, the server uses the pre-processed data to perform analysis with an AI model that employs machine learning libraries such as TENSORFLOW®. This AI model has already learned the normal lifestyle patterns of elderly people from past data and sets criteria for detecting abnormalities based on this.

[0115] If an anomaly is detected, the server uses Firebase to send a notification to the user's mobile device. This allows family members and caregivers to immediately check the situation and take necessary action.

[0116] This system also periodically adjusts the parameters of the AI ​​model based on evaluations and feedback from users. This ensures that the accuracy of the analysis is continuously improved.

[0117] For example, if water usage occurs at an unusual time during a family trip, the system will immediately detect this anomaly and send an alert to a mobile device. The user can then receive the notification and check the safety of their home.

[0118] Examples of prompt statements are as follows:

[0119] "Detect any anomalies in the electricity and water usage pattern data for the past 7 days. If there are deviations from normal conditions, analyze the reasons and create a report."

[0120] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0121] Step 1:

[0122] Multiple sensors installed in the home collect electricity and water usage data in real time. The system receives data from each sensor as input. This includes electricity meters, flow sensors, and acoustic sensors. The collected data is immediately transmitted to a server.

[0123] Step 2:

[0124] The server performs noise reduction and imputation on the received sensor data. The input is raw data from the sensor, and the output is a clean dataset with noise removed and missing values ​​imputed. Specifically, it filters out inconsistent data points and applies imputation to maintain the continuity of the time-series data.

[0125] Step 3:

[0126] The server inputs preprocessed data into a machine learning model using TensorFlow and performs analysis. The input is a clean dataset, and the output is the user's everyday behavior patterns. The machine learning model has already learned patterns based on past data and generates a behavioral model to build criteria for anomaly detection.

[0127] Step 4:

[0128] Based on the analysis results, the server sets thresholds for detecting anomalies and continues monitoring. The input is the behavioral patterns from the machine learning model, and the output is the threshold for anomaly detection. The server continuously evaluates whether there are any signs of anomaly.

[0129] Step 5:

[0130] When an anomaly is detected, the server uses Firebase to instruct the system to send an alert to the user's mobile device. The input is the result of the anomaly detection, and the output is the alert notification. The server sends an emergency alert to the user's device, prompting immediate action on the problem.

[0131] Step 6:

[0132] The user or their family member will check the details of the anomaly through the terminal and take appropriate action as needed. The input is an alarm notification, and the output is the user's verification of the anomaly and the corresponding actions taken. Based on the received notification, the user will check the safety of the elderly person and, if necessary, contact other relevant parties.

[0133] Step 7:

[0134] User feedback is collected, and the server uses this feedback to adjust the machine learning model. The input is the user feedback, and the output is the parameters of the adjusted model. This allows the system to improve the accuracy of analysis and anomaly detection.

[0135] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0136] This invention provides a system that enables monitoring of household electricity, water, and gas usage, as well as emotional states based on sound and video. It analyzes this data to monitor the user's lifestyle and emotional patterns. The operation of this system is described in detail below.

[0137] First, the various sensors installed in the user's home include power consumption meters, flow sensors, sound sensors, and cameras that acquire visual data. These sensors collect data on daily life within the home, and the data is transmitted to a server via a terminal.

[0138] The server performs noise reduction and data imputation on electricity, water, and gas data to generate standardized datasets. Acoustic and video data undergo similar preprocessing. This results in clean and consistent data.

[0139] Next, the server utilizes an artificial intelligence (AI) analysis module to analyze lifestyle patterns and emotional states based on this data. In analyzing lifestyle patterns, it learns a standard behavioral timetable based on conventional electricity and water usage and detects anomalies. The emotion engine analyzes information obtained from audio and video to recognize emotions from the user's facial expressions and tone of voice, and stores these as daily emotional patterns.

[0140] Using the data collected in this way, the server can monitor not only abnormal lifestyle patterns but also abnormal emotional states. If an anomaly is detected, the server sends an alert to a pre-designated recipient, such as a family member or caregiver. Specifically, if usage patterns deviate significantly from normal lifestyle patterns or if it is determined that the user is experiencing high levels of emotional stress, the server will promptly notify the user accordingly.

[0141] Furthermore, feedback from users and recipients is collected, and the AI ​​model is adjusted to improve the accuracy of the analysis. This feedback loop allows the system to be continuously optimized, providing a natural monitoring environment that does not require elderly people to perform specific actions.

[0142] The following describes the processing flow.

[0143] Step 1:

[0144] Various sensors and cameras installed in the user's home are operating, continuously measuring electricity, water, and gas usage, as well as acoustic and visual data. The data is collected on a terminal in real time.

[0145] Step 2:

[0146] The terminal formats the collected data and sends it to the server via the communication line. This process minimizes data loss and enables smooth data transmission and reception.

[0147] Step 3:

[0148] The server denoises the received data and fills in any missing data based on past data. At this stage, consistency processing is performed to improve the reliability of the data.

[0149] Step 4:

[0150] The server inputs pre-processed data into an artificial intelligence model to analyze the user's daily life patterns and emotional patterns. Numerical data such as the frequency of electricity, water, and gas usage is plotted on a timeline to learn the user's lifestyle habits. Acoustic and visual data are used to determine the user's emotional state using facial recognition and voice analysis, and this is compared to normal conditions.

[0151] Step 5:

[0152] The server performs an anomaly monitoring process based on the lifestyle and emotional patterns obtained from the analysis. Here, deviations from the normal are determined based on defined thresholds and rules. For example, if usage frequency drops significantly or emotional stress is assessed as high, it will be treated as an anomaly.

[0153] Step 6:

[0154] If an anomaly is detected, the server generates an alarm and sends a notification to pre-registered family members or caregivers via the terminal. The notification includes the detected anomaly and its details, allowing recipients to consider immediate action.

[0155] Step 7:

[0156] Users or recipients report false positives and areas for system improvement through an interface designed to collect feedback. This feedback is recorded by the server and used to readjust the AI ​​model.

[0157] Step 8:

[0158] Based on feedback, the server optimizes the parameters of the artificial intelligence model, improving the accuracy of anomaly detection and alert transmission. This allows the system to continuously evolve and support a safe and secure living environment for the elderly.

[0159] (Example 2)

[0160] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0161] Conventional home monitoring systems only track electricity, water, and gas usage, making it difficult to monitor users' lifestyles and emotional states in detail. Furthermore, they lacked the ability to detect anomalies early and provide appropriate feedback, making it challenging to effectively support users' health and safety. Therefore, there is a need for a system that comprehensively analyzes overall household data and provides rapid and appropriate notification of anomalies.

[0162] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0163] In this invention, the server includes means for acquiring indoor resource consumption and environmental data, means for applying removal filters and complementary algorithms, and means for analyzing the data using an intelligent analysis engine and learning individual behavioral patterns. This makes it possible to analyze lifestyle patterns and emotional states with high accuracy, quickly detect anomalies, and send warnings.

[0164] A "detection device" refers to an instrument or sensor used to measure the state of the environment or an object, and includes, for example, power consumption meters, flow sensors, sound detection sensors, and cameras.

[0165] "Resource consumption" refers to the consumption of energy and resources such as electricity, water, and gas used indoors.

[0166] "Environmental data" refers to information such as sound and video in a specific space, including acoustic and visual information.

[0167] A "removal filter" refers to an algorithm used to remove unnecessary information, such as noise, from data.

[0168] A "complementary algorithm" refers to a method for inferring or reconstructing missing parts of collected data.

[0169] An "intelligent analysis engine" refers to artificial intelligence technology that highly analyzes collected data and identifies patterns.

[0170] "Behavioral patterns" refer to a timeline based on past data that shows an individual's lifestyle habits and tendencies in daily behavior.

[0171] The system for realizing this invention enables detailed monitoring of resource consumption and environmental data within the home. The main hardware consists of multiple sensor groups and a data processing unit. Specifically, it includes a power consumption meter, flow sensors, sound sensors, and cameras. These sensors are responsible for acquiring data on electricity, water, and gas consumption within the home, as well as audio and video data.

[0172] The user sends the collected data to the server via their device. The device acts as a temporary relay point for the data, ensuring communication stability. After transmission, the data is received by the server and preprocessed using denoising filters and imputation algorithms. This process removes noise and infers and imputes missing data, resulting in a clean and consistent dataset.

[0173] Next, the server starts up the intelligent analysis engine and analyzes the pre-processed data. Generative AI models are used for the analysis, which are capable of precisely learning individual behavioral patterns and emotional states. For example, this includes methods for learning normal behavioral patterns from power usage data, and methods for recognizing emotions from voice tone and facial expressions. Anomaly detection is then performed using the data analyzed in this way.

[0174] If an anomaly is detected, the server immediately sends an alert to the terminals of pre-configured recipients (e.g., family members or medical personnel). This is a crucial function for continuously monitoring the user's health and safety. Furthermore, feedback from users and recipients is collected and used to refine the accuracy of the intelligent analysis engine. This process automatically optimizes the system, enabling more effective monitoring.

[0175] A concrete example of a prompt message could be, "Explain a mechanism that detects anomalies from a user's daily life data and notifies the user of their emotional state when stress levels are high." By entering this prompt message into the system, it becomes possible to discuss and deepen understanding of various aspects of the user's life.

[0176] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0177] Step 1:

[0178] Within the user's home, various sensors acquire power consumption, flow rate, audio, and video data in real time. This creates a detailed dataset of the user's daily life. The input is raw data transmitted from the sensors, and the output is data obtained in a format that is stored in temporary data storage.

[0179] Step 2:

[0180] The terminal receives data collected from the sensor and performs an initial integrity check. It identifies missing or inconsistent data and makes any necessary adjustments to ensure data reliability. Raw data from the sensor is used as input, and pre-processed data is obtained as output.

[0181] Step 3:

[0182] The server receives data sent from the terminal and applies denoising filters and interpolation algorithms. By reducing noisy data and imputing missing parts, it generates a clean and complete dataset. The input is the initial data sent from the terminal, and the output is the denoised and interpolated data.

[0183] Step 4:

[0184] The server analyzes pre-processed data using a generation AI model. It analyzes power usage patterns, voice tone, and facial expression data to learn lifestyle patterns and emotional states. The input is a clean dataset, and the output generates a model of the user's behavioral patterns and emotional states.

[0185] Step 5:

[0186] The server performs anomaly detection based on the generated model. It compares the analysis results with pre-set criteria and thresholds to identify behaviors and emotional states that deviate from normal patterns. Behavioral patterns and emotional state models are used as input, and the anomaly determination result is obtained as output.

[0187] Step 6:

[0188] The server sends an alert to pre-configured recipients if an anomaly is detected. The recipients' devices receive notifications, providing alerts regarding safety and health management. The input is the anomaly detection result, and the output is the notification sent to the recipient's device.

[0189] Step 7:

[0190] Users and recipients provide feedback to the system. Based on this feedback, the generated AI model is recalibrated, improving the accuracy of the analysis. Feedback information is used as input, and the improved model is obtained as output.

[0191] (Application Example 2)

[0192] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0193] In systems aimed at improving safety and security in daily life by comprehensively monitoring energy, resource, and water consumption within the home, as well as the emotional state of residents, and quickly detecting and notifying users of anomalies, conventional systems have the challenge of handling individual data separately, making unified monitoring difficult. Furthermore, there are concerns that delays in anomaly detection could lead to inadequate responses in situations requiring rapid action. In addition, there is a need to detect changes in emotional state early and enable appropriate feedback.

[0194] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0195] In this invention, the server includes means for acquiring energy, water, and fuel usage in a living space via one or more detection devices; means for preprocessing the acquired data by removing unwanted components and imputing missing values; and means for analyzing the preprocessed data using a machine learning model to learn the user's activity patterns. This makes it possible to detect abnormalities in the resident's emotional state and activity patterns in real time and promptly notify the resident's mobile communication terminal.

[0196] A "detection device" is a device used to measure the usage of electricity, water, and fuel within a living space. This includes sensors and meters.

[0197] "Removal of unwanted components" is a data preprocessing operation that removes noise and outliers contained within the data.

[0198] "Missing value imputation" is the operation of filling in missing values ​​in an incomplete dataset to generate complete data suitable for analysis.

[0199] A "machine learning model" is a set of algorithms used to learn patterns from data and perform analysis and predictions.

[0200] "Activity patterns" refer to a series of actions and tendencies related to a user's energy and resource consumption.

[0201] An "abnormality" is a sign that indicates behavior or a situation that deviates from normal activity patterns, and often requires a prompt response.

[0202] "Real-time notification" is a process that immediately transmits information about analyzed anomalies and emotional states to the user's mobile communication terminal.

[0203] The system implementing this invention centrally monitors the consumption of electricity, water, and fuel within the home, analyzes the emotional state of the residents to detect anomalies, and provides prompt notification. The server operates this system in the following specific manner.

[0204] First, the server collects energy and resource usage data from multiple detection devices. These include electricity meters, flow sensors, and gas detectors. These devices record specific consumption patterns within the living space in real time and transmit the data to the server.

[0205] Next, the server removes unwanted components and imputes missing values ​​from the received data to generate a clean dataset suitable for analysis. Specifically, it processes noise and anomalous missing data, transforming it into complete and consistent data.

[0206] Subsequently, the server operates a machine learning model based on the prepared data to learn the activity patterns of the residents. This model typically uses open-source libraries such as TensorFlow or PyTorch. The machine learning model analyzes past consumption data and acoustic information to detect stable behavioral patterns and emotional changes.

[0207] If an anomaly is detected, for example, if water usage increases significantly more than usual and acoustic data indicates feelings of anxiety, the server immediately activates a real-time notification function. This allows the user's mobile device to be notified of the anomaly. This notification is sent via a communication platform such as Firebase Cloud Messaging.

[0208] For example, if water leakage continues for an extended period, or if stress or anxiety is detected from the user's voice, a prompt message such as "A deviation from your normal lifestyle has been detected. Please check immediately." will be generated and sent as a notification.

[0209] In this way, the server functions as a system that enhances the safety and sense of security in the living space.

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The server collects data from various detection devices installed in the home. Specifically, it acquires data in real time from devices such as power meters, flow sensors, and gas meters, and transmits it to the server. The input is raw data from these devices, and the output is unprocessed data aggregated on the server.

[0213] Step 2:

[0214] The server performs de-noise removal on the collected data. This process filters out data that may contain noise or outliers, generating a clean dataset. For example, if a sudden spike in power consumption is temporary noise, this is corrected. The input is the raw data from step 1, and the output is the de-noise-removed data.

[0215] Step 3:

[0216] The server performs missing value imputation on the data from Step 2. This step fills in data that was intermittently unavailable both temporally and correlatively. Artificial intelligence is used to imputate incomplete data based on historical data and trends. The input is clean data with missing values, and the output is a complete dataset.

[0217] Step 4:

[0218] The server uses a machine learning model to analyze the data from step 3. Here, it learns user activity patterns and emotional states to form standard patterns. Libraries such as TensorFlow and PyTorch are used for this analysis. The input is the complete dataset, and the output is the analyzed pattern information.

[0219] Step 5:

[0220] The server performs anomaly detection based on the analyzed pattern information. It sets specific thresholds and evaluates the data against them to identify anomalies. For example, this might apply if water usage significantly exceeds normal levels. The input consists of the analyzed pattern information and current data, while the output indicates whether an anomaly exists and provides detailed information about it.

[0221] Step 6:

[0222] If an anomaly is detected, the server immediately sends a real-time notification to the user's mobile device. This notification is sent using Firebase Cloud Messaging, etc. The prompt message sent will be, "A deviation from your normal daily routine has been detected. Please check immediately." The input is the anomaly detection information, and the output is the notification sent to the user's device.

[0223] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0224] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0225] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0226] [Second Embodiment]

[0227] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0228] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0229] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0230] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0231] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0232] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0233] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0234] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0235] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0236] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0237] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0238] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0239] This invention is a system for monitoring the living conditions of elderly people. It combines machine learning technology to analyze the usage of electricity, water, and gas within the home, monitors living patterns, and detects abnormalities. The operation flow of this system is described below.

[0240] First, several types of sensors are installed in the user's home. These sensors include power consumption meters, flow sensors, and voice sensors. These sensors monitor the use of electricity, gas, and water in the home in real time.

[0241] Data obtained from each sensor is transmitted to a server via the network through a terminal. The server removes noise from the received data and appropriately imputes missing data. This preprocessing is performed to improve the accuracy of the analysis.

[0242] Next, the server inputs the pre-processed data into an artificial intelligence model for analysis. This AI model learns the user's lifestyle patterns based on past data and builds a model of normal behavior. For example, it can detect when electricity consumption increases each morning and recognize the user's wake-up pattern.

[0243] Based on established usage patterns, the server sets criteria for detecting anomalies. These include deviations from normal usage patterns and periods of absence from use. When an anomaly is detected, the server sends an alert to the user's family or caregiver's device.

[0244] As a concrete example, suppose an elderly person has a routine of using an electric kettle at 8 AM every morning, followed by a shower. If this routine suddenly stops completely, the server will determine it's an anomaly based on defined criteria and send an alert. The user's family can receive a notification on their smartphone and quickly check the situation.

[0245] Finally, based on user feedback, the server continuously refines its machine learning model to further improve the system's accuracy. This approach enables continuous monitoring without requiring elderly individuals to operate specific technological devices.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] Various sensors installed in the user's home measure the usage of electricity, water, gas, and voice, and transmit the data to a terminal.

[0249] Step 2:

[0250] The terminal converts the collected data into a format suitable for transfer to the server and sends it to the server via the network.

[0251] Step 3:

[0252] The server performs noise reduction processing and handles missing values ​​on the received raw data. Specifically, it cleans up the data using appropriate filtering techniques.

[0253] Step 4:

[0254] The server inputs pre-processed data into an artificial intelligence (AI) analysis unit, uses a trained model to analyze the user's daily life patterns, and constructs a baseline daily rhythm.

[0255] Step 5:

[0256] The server sets anomaly detection criteria based on the established lifestyle rhythm, and monitors for abnormal patterns while performing real-time data analysis.

[0257] Step 6:

[0258] When an anomaly is detected, for example, if there is no usage for an extended period during normal operating hours, the server generates an alert and prepares to send a warning to pre-registered recipients.

[0259] Step 7:

[0260] The device (for example, the user's family member or caregiver's smartphone) receives an alert notification from the server and immediately displays the notification to the user. This allows them to check the situation.

[0261] Step 8:

[0262] If a false positive occurs based on user feedback, the server receives the feedback data, adjusts the AI ​​model, and improves the accuracy of subsequent analyses.

[0263] (Example 1)

[0264] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] There is a need to provide a safe and secure living environment for the elderly and those requiring care, but conventional monitoring systems lack the accuracy to detect abnormal behavior in real time. Furthermore, it is difficult to conduct unbiased monitoring while protecting user privacy. It is necessary to solve these problems and realize a safe and efficient monitoring system.

[0266] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0267] In this invention, the server includes means for acquiring the usage status of energy resources in the environment via one or more sensing devices, means for performing noise reduction and missing information imputation for preprocessing the acquired information, and means for analyzing the preprocessed information using a machine learning algorithm to learn the behavioral patterns of the person being monitored. This enables high-precision monitoring of the person's lifestyle and allows for a quick and appropriate response when an anomaly is detected. Furthermore, it enables continuous monitoring while protecting privacy.

[0268] A "sensing device" is a device used to measure the usage of energy resources such as electricity, water, and gas in the environment and to acquire that information.

[0269] "Information preprocessing" is a process that improves data quality by removing noise and filling in missing information from acquired data.

[0270] A "machine learning algorithm" is a computational method that identifies patterns from large amounts of data and learns from them to make predictions and classifications about future data.

[0271] A "behavioral pattern" is a set of characteristics that indicate the regularity or tendencies of the daily activities and behaviors of the person being monitored.

[0272] An "abnormal pattern" refers to irregular activity or behavior that differs from the normal behavioral patterns learned by a machine learning algorithm.

[0273] A "criterion or threshold" is a criterion or limit used to make a judgment when detecting an unusual pattern.

[0274] A "warning signal" is an alarm message sent to a pre-configured recipient when an unusual pattern is detected.

[0275] A "mobile communication device" is a portable device that a user can carry and use to receive communications and notifications from external sources.

[0276] This invention is a monitoring system primarily for safely observing the lives of the elderly and those requiring care. The system operates as follows:

[0277] Multiple sensing devices are installed in the user's environment. These devices measure the usage of energy resources such as electricity, water, and gas in real time. This allows for detailed monitoring of usage patterns in daily life.

[0278] The collected information is transmitted to the server via the terminal. Upon arrival at the server, the information undergoes preprocessing, including noise reduction and the imputation of missing information. This preprocessing improves the accuracy of the data, enabling more reliable analysis.

[0279] The preprocessed data is then analyzed using machine learning algorithms. The server utilizes this algorithm to learn the behavior patterns of the monitored person. As a specific example, it is possible to grasp the pattern that the monitored person uses an electric kettle at a specific time every day.

[0280] Based on the analysis results, the server sets criteria and thresholds for detecting abnormal behavior. Through this process, it is possible to quickly detect when behavior different from normal occurs.

[0281] When an abnormality is detected, the server sends a warning signal to a pre-set recipient. For example, when the monitored person stops using all the electrical appliances they usually use, an alert can be quickly sent to the family. Since this warning signal is delivered as a notification to the user's mobile communication device, it is possible to respond promptly.

[0282] Also, by leveraging feedback from the user and family, the server continuously adjusts the parameters of the machine learning algorithm to improve the accuracy of the system. Through this dynamic adjustment, the system can flexibly respond to changing environments and new behavior patterns.

[0283] As described above, this system uses a generative AI model to achieve effective monitoring. As an example of a prompt sentence, by inputting content such as "How can we learn the living patterns of the elderly and improve the method of notifying abnormalities?" into the generative AI model, more efficient solutions can be found.

[0284] The flow of the specific process in Example 1 will be described using FIG. 11.

[0285] Step 1:

[0286] The user installs multiple sensing devices at home. These devices collect data on the usage of electricity, water, and gas in real-time. The input is the usage data of various energy resources, and the output is the usage pattern of the surrounding environment over time. This process enables detailed monitoring of the residents' lives.

[0287] Step 2:

[0288] The terminal sends the collected data to the server via the network. The input is the raw data from the sensing device, and the output is secure data packets reaching the server. Here, data encryption technology is used to ensure the security of information and prevent unauthorized access from outside.

[0289] Step 3:

[0290] The server preprocesses the received data. Specifically, a noise removal filter is applied, and missing data is complemented based on statistical methods. The input is the raw data sent from the terminal, and the output is a reliable clean dataset. This process enhances the analysis performance.

[0291] Step 4:

[0292] The server inputs the preprocessed data into a machine learning model for analysis. The algorithm learns the repeated data and extracts the normal behavior patterns of the user. The input is the clean dataset, and the output is the behavior pattern of the monitored person generated. Through this analysis, daily habits are identified.

[0293] Step 5:

[0294] The server sets anomaly detection criteria based on the analysis results. A threshold is set, and operations exceeding it are regarded as anomalies. The input is the behavior pattern data, and the output is the formulation of anomaly detection rules. This enables the immediate detection of operations outside the normal range.

[0295] Step 6:

[0296] If an anomaly is detected, the server sends a warning signal. An alarm is sent to the terminal of a pre-configured recipient. The input is the result of the anomaly detection algorithm, and the output is an alert notification sent to the recipient. This notification prompts a quick response.

[0297] Step 7:

[0298] The user or their representative sends feedback to the server. The server uses this feedback to adjust the parameters of the machine learning model, improving the system's accuracy. The input is the feedback information, and the output is the optimized model parameters. This adjustment leads to continuous improvement.

[0299] (Application Example 1)

[0300] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0301] There is a need to efficiently and continuously detect abnormalities within the home and respond quickly while ensuring the safety of the elderly. Conventional systems only monitor electricity and water usage, making it difficult to immediately detect unauthorized access from external sources or suspicious usage patterns. This invention aims to solve these problems and enhance the safety of the home and its surrounding environment.

[0302] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0303] In this invention, the server includes means for acquiring the usage status of energy and liquids in the home via one or more detection means, means for performing noise removal and missing value processing for preprocessing the acquired information, means for analyzing the preprocessed information using a machine learning model to learn the user's behavior pattern, and means for detecting unauthorized use from the outside and recording and notifying of the abnormality. Thereby, potential dangers faced by the elderly in the home can be discovered early and prompt response becomes possible.

[0304] The "detection means" is a device or group of devices installed to grasp the usage status of energy and liquids in the home.

[0305] "Preprocessing" is the process of removing noise from the acquired information and supplementing missing values.

[0306] The "machine learning model" is an artificial intelligence-based algorithm used to learn and analyze the user's behavior pattern from the preprocessed information.

[0307] The "rule or criterion" is a rule or threshold set to detect an abnormal pattern based on the analyzed information.

[0308] The "communication means" is a means for transmitting an alarm to a pre-set recipient when an abnormal pattern is detected.

[0309] "Evaluation" is opinions or information regarding the operation of the system received as feedback from the user.

[0310] The "condition" is the parameter or setting of the model that is changed when adjusting the machine learning model.

[0311] "Detection of unauthorized use" is the process of discovering unauthorized use or access from the outside in or around the home.

[0312] "Energy and liquids" refers to infrastructure resources such as electricity and water consumed within the home.

[0313] The system for implementing this invention has the function of monitoring and analyzing household electricity and water usage to detect anomalies in order to ensure the safety of the elderly. This system is operated using a server, terminals, a group of sensors, and a machine learning model.

[0314] The server receives data acquired from multiple sensors installed within the home, including power consumption meters and flow sensors. The received data undergoes noise reduction and interpolation on the server, and is pre-processed for analysis.

[0315] Next, the server uses the pre-processed data to perform analysis with an AI model that employs machine learning libraries such as TensorFlow. This AI model has already learned the normal lifestyle patterns of elderly people from past data and sets criteria for detecting anomalies based on this.

[0316] If an anomaly is detected, the server uses Firebase to send a notification to the user's mobile device. This allows family members and caregivers to immediately check the situation and take necessary action.

[0317] This system also periodically adjusts the parameters of the AI ​​model based on evaluations and feedback from users. This ensures that the accuracy of the analysis is continuously improved.

[0318] For example, if water usage occurs at an unusual time during a family trip, the system will immediately detect this anomaly and send an alert to a mobile device. The user can then receive the notification and check the safety of their home.

[0319] Examples of prompt statements are as follows:

[0320] "Detect any anomalies in the electricity and water usage pattern data for the past 7 days. If there are deviations from normal conditions, analyze the reasons and create a report."

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] Multiple sensors installed in the home collect electricity and water usage data in real time. The system receives data from each sensor as input. This includes electricity meters, flow sensors, and acoustic sensors. The collected data is immediately transmitted to a server.

[0324] Step 2:

[0325] The server performs noise reduction and imputation on the received sensor data. The input is raw data from the sensor, and the output is a clean dataset with noise removed and missing values ​​imputed. Specifically, it filters out inconsistent data points and applies imputation to maintain the continuity of the time-series data.

[0326] Step 3:

[0327] The server inputs preprocessed data into a machine learning model using TensorFlow and performs analysis. The input is a clean dataset, and the output is the user's everyday behavior patterns. The machine learning model has already learned patterns based on past data and generates a behavioral model to build criteria for anomaly detection.

[0328] Step 4:

[0329] Based on the analysis results, the server sets thresholds for detecting anomalies and continues monitoring. The input is the behavioral patterns from the machine learning model, and the output is the threshold for anomaly detection. The server continuously evaluates whether there are any signs of anomaly.

[0330] Step 5:

[0331] When an anomaly is detected, the server uses Firebase to instruct the system to send an alert to the user's mobile device. The input is the result of the anomaly detection, and the output is the alert notification. The server sends an emergency alert to the user's device, prompting immediate action on the problem.

[0332] Step 6:

[0333] The user or their family member will check the details of the anomaly through the terminal and take appropriate action as needed. The input is an alarm notification, and the output is the user's verification of the anomaly and the corresponding actions taken. Based on the received notification, the user will check the safety of the elderly person and, if necessary, contact other relevant parties.

[0334] Step 7:

[0335] User feedback is collected, and the server uses this feedback to adjust the machine learning model. The input is the user feedback, and the output is the parameters of the adjusted model. This allows the system to improve the accuracy of analysis and anomaly detection.

[0336] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0337] This invention provides a system that enables monitoring of household electricity, water, and gas usage, as well as emotional states based on sound and video. It analyzes this data to monitor the user's lifestyle and emotional patterns. The operation of this system is described in detail below.

[0338] First, the various sensors installed in the user's home include power consumption meters, flow sensors, sound sensors, and cameras that acquire visual data. These sensors collect data on daily life within the home, and the data is transmitted to a server via a terminal.

[0339] The server performs noise reduction and data imputation on electricity, water, and gas data to generate standardized datasets. Acoustic and video data undergo similar preprocessing. This results in clean and consistent data.

[0340] Next, the server utilizes an artificial intelligence (AI) analysis module to analyze lifestyle patterns and emotional states based on this data. In analyzing lifestyle patterns, it learns a standard behavioral timetable based on conventional electricity and water usage and detects anomalies. The emotion engine analyzes information obtained from audio and video to recognize emotions from the user's facial expressions and tone of voice, and stores these as daily emotional patterns.

[0341] Using the data collected in this way, the server can monitor not only abnormal lifestyle patterns but also abnormal emotional states. If an anomaly is detected, the server sends an alert to a pre-designated recipient, such as a family member or caregiver. Specifically, if usage patterns deviate significantly from normal lifestyle patterns or if it is determined that the user is experiencing high levels of emotional stress, the server will promptly notify the user accordingly.

[0342] Furthermore, feedback from users and recipients is collected, and the AI ​​model is adjusted to improve the accuracy of the analysis. This feedback loop allows the system to be continuously optimized, providing a natural monitoring environment that does not require elderly people to perform specific actions.

[0343] The following describes the processing flow.

[0344] Step 1:

[0345] Various sensors and cameras installed in the user's home are operating, continuously measuring electricity, water, and gas usage, as well as acoustic and visual data. The data is collected on a terminal in real time.

[0346] Step 2:

[0347] The terminal formats the collected data and sends it to the server via the communication line. This process minimizes data loss and enables smooth data transmission and reception.

[0348] Step 3:

[0349] The server denoises the received data and fills in any missing data based on past data. At this stage, consistency processing is performed to improve the reliability of the data.

[0350] Step 4:

[0351] The server inputs pre-processed data into an artificial intelligence model to analyze the user's daily life patterns and emotional patterns. Numerical data such as the frequency of electricity, water, and gas usage is plotted on a timeline to learn the user's lifestyle habits. Acoustic and visual data are used to determine the user's emotional state using facial recognition and voice analysis, and this is compared to normal conditions.

[0352] Step 5:

[0353] The server performs an anomaly monitoring process based on the lifestyle and emotional patterns obtained from the analysis. Here, deviations from the normal are determined based on defined thresholds and rules. For example, if usage frequency drops significantly or emotional stress is assessed as high, it will be treated as an anomaly.

[0354] Step 6:

[0355] If an anomaly is detected, the server generates an alarm and sends a notification to pre-registered family members or caregivers via the terminal. The notification includes the detected anomaly and its details, allowing recipients to consider immediate action.

[0356] Step 7:

[0357] Users or recipients report false positives and areas for system improvement through an interface designed to collect feedback. This feedback is recorded by the server and used to readjust the AI ​​model.

[0358] Step 8:

[0359] Based on feedback, the server optimizes the parameters of the artificial intelligence model, improving the accuracy of anomaly detection and alert transmission. This allows the system to continuously evolve and support a safe and secure living environment for the elderly.

[0360] (Example 2)

[0361] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0362] Conventional home monitoring systems only track electricity, water, and gas usage, making it difficult to monitor users' lifestyles and emotional states in detail. Furthermore, they lacked the ability to detect anomalies early and provide appropriate feedback, making it challenging to effectively support users' health and safety. Therefore, there is a need for a system that comprehensively analyzes overall household data and provides rapid and appropriate notification of anomalies.

[0363] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0364] In this invention, the server includes means for acquiring indoor resource consumption and environmental data, means for applying removal filters and complementary algorithms, and means for analyzing the data using an intelligent analysis engine and learning individual behavioral patterns. This makes it possible to analyze lifestyle patterns and emotional states with high accuracy, quickly detect anomalies, and send warnings.

[0365] A "detection device" refers to an instrument or sensor used to measure the state of the environment or an object, and includes, for example, power consumption meters, flow sensors, sound detection sensors, and cameras.

[0366] "Resource consumption" refers to the consumption of energy and resources such as electricity, water, and gas used indoors.

[0367] "Environmental data" refers to information such as sound and video in a specific space, including acoustic and visual information.

[0368] A "removal filter" refers to an algorithm used to remove unnecessary information, such as noise, from data.

[0369] A "complementary algorithm" refers to a method for inferring or reconstructing missing parts of collected data.

[0370] An "intelligent analysis engine" refers to artificial intelligence technology that highly analyzes collected data and identifies patterns.

[0371] "Behavioral patterns" refer to a timeline based on past data that shows an individual's lifestyle habits and tendencies in daily behavior.

[0372] The system for realizing this invention enables detailed monitoring of resource consumption and environmental data within the home. The main hardware consists of multiple sensor groups and a data processing unit. Specifically, it includes a power consumption meter, flow sensors, sound sensors, and cameras. These sensors are responsible for acquiring data on electricity, water, and gas consumption within the home, as well as audio and video data.

[0373] The user sends the collected data to the server via their device. The device acts as a temporary relay point for the data, ensuring communication stability. After transmission, the data is received by the server and preprocessed using denoising filters and imputation algorithms. This process removes noise and infers and imputes missing data, resulting in a clean and consistent dataset.

[0374] Next, the server starts up the intelligent analysis engine and analyzes the pre-processed data. Generative AI models are used for the analysis, which are capable of precisely learning individual behavioral patterns and emotional states. For example, this includes methods for learning normal behavioral patterns from power usage data, and methods for recognizing emotions from voice tone and facial expressions. Anomaly detection is then performed using the data analyzed in this way.

[0375] If an anomaly is detected, the server immediately sends an alert to the terminals of pre-configured recipients (e.g., family members or medical personnel). This is a crucial function for continuously monitoring the user's health and safety. Furthermore, feedback from users and recipients is collected and used to refine the accuracy of the intelligent analysis engine. This process automatically optimizes the system, enabling more effective monitoring.

[0376] A concrete example of a prompt message could be, "Explain a mechanism that detects anomalies from a user's daily life data and notifies the user of their emotional state when stress levels are high." By entering this prompt message into the system, it becomes possible to discuss and deepen understanding of various aspects of the user's life.

[0377] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0378] Step 1:

[0379] Within the user's home, various sensors acquire power consumption, flow rate, audio, and video data in real time. This creates a detailed dataset of the user's daily life. The input is raw data transmitted from the sensors, and the output is data obtained in a format that is stored in temporary data storage.

[0380] Step 2:

[0381] The terminal receives data collected from the sensor and performs an initial integrity check. It identifies missing or inconsistent data and makes any necessary adjustments to ensure data reliability. Raw data from the sensor is used as input, and pre-processed data is obtained as output.

[0382] Step 3:

[0383] The server receives data sent from the terminal and applies denoising filters and interpolation algorithms. By reducing noisy data and imputing missing parts, it generates a clean and complete dataset. The input is the initial data sent from the terminal, and the output is the denoised and interpolated data.

[0384] Step 4:

[0385] The server analyzes pre-processed data using a generation AI model. It analyzes power usage patterns, voice tone, and facial expression data to learn lifestyle patterns and emotional states. The input is a clean dataset, and the output generates a model of the user's behavioral patterns and emotional states.

[0386] Step 5:

[0387] The server performs anomaly detection based on the generated model. It compares the analysis results with pre-set criteria and thresholds to identify behaviors and emotional states that deviate from normal patterns. Behavioral patterns and emotional state models are used as input, and the anomaly determination result is obtained as output.

[0388] Step 6:

[0389] The server sends an alert to pre-configured recipients if an anomaly is detected. The recipients' devices receive notifications, providing alerts regarding safety and health management. The input is the anomaly detection result, and the output is the notification sent to the recipient's device.

[0390] Step 7:

[0391] Users and recipients provide feedback to the system. Based on this feedback, the generated AI model is recalibrated, improving the accuracy of the analysis. Feedback information is used as input, and the improved model is obtained as output.

[0392] (Application Example 2)

[0393] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0394] In systems aimed at improving safety and security in daily life by comprehensively monitoring energy, resource, and water consumption within the home, as well as the emotional state of residents, and quickly detecting and notifying users of anomalies, conventional systems have the challenge of handling individual data separately, making unified monitoring difficult. Furthermore, there are concerns that delays in anomaly detection could lead to inadequate responses in situations requiring rapid action. In addition, there is a need to detect changes in emotional state early and enable appropriate feedback.

[0395] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0396] In this invention, the server includes means for acquiring energy, water, and fuel usage in a living space via one or more detection devices; means for preprocessing the acquired data by removing unwanted components and imputing missing values; and means for analyzing the preprocessed data using a machine learning model to learn the user's activity patterns. This makes it possible to detect abnormalities in the resident's emotional state and activity patterns in real time and promptly notify the resident's mobile communication terminal.

[0397] A "detection device" is a device used to measure the usage of electricity, water, and fuel within a living space. This includes sensors and meters.

[0398] "Removal of unwanted components" is a data preprocessing operation that removes noise and outliers contained within the data.

[0399] "Missing value imputation" is the operation of filling in missing values ​​in an incomplete dataset to generate complete data suitable for analysis.

[0400] A "machine learning model" is a set of algorithms used to learn patterns from data and perform analysis and predictions.

[0401] "Activity patterns" refer to a series of actions and tendencies related to a user's energy and resource consumption.

[0402] An "abnormality" is a sign that indicates behavior or a situation that deviates from normal activity patterns, and often requires a prompt response.

[0403] "Real-time notification" is a process that immediately transmits information about analyzed anomalies and emotional states to the user's mobile communication terminal.

[0404] The system implementing this invention centrally monitors the consumption of electricity, water, and fuel within the home, analyzes the emotional state of the residents to detect anomalies, and provides prompt notification. The server operates this system in the following specific manner.

[0405] First, the server collects energy and resource usage data from multiple detection devices. These include electricity meters, flow sensors, and gas detectors. These devices record specific consumption patterns within the living space in real time and transmit the data to the server.

[0406] Next, the server removes unwanted components and imputes missing values ​​from the received data to generate a clean dataset suitable for analysis. Specifically, it processes noise and anomalous missing data, transforming it into complete and consistent data.

[0407] Subsequently, the server operates a machine learning model based on the prepared data to learn the activity patterns of the residents. This model typically uses open-source libraries such as TensorFlow or PyTorch. The machine learning model analyzes past consumption data and acoustic information to detect stable behavioral patterns and emotional changes.

[0408] If an anomaly is detected, for example, if water usage increases significantly more than usual and acoustic data indicates feelings of anxiety, the server immediately activates a real-time notification function. This allows the user's mobile device to be notified of the anomaly. This notification is sent via a communication platform such as Firebase Cloud Messaging.

[0409] For example, if water leakage continues for an extended period, or if stress or anxiety is detected from the user's voice, a prompt message such as "A deviation from your normal lifestyle has been detected. Please check immediately." will be generated and sent as a notification.

[0410] In this way, the server functions as a system that enhances the safety and sense of security in the living space.

[0411] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0412] Step 1:

[0413] The server collects data from various detection devices installed in the home. Specifically, it acquires data in real time from devices such as power meters, flow sensors, and gas meters, and transmits it to the server. The input is raw data from these devices, and the output is unprocessed data aggregated on the server.

[0414] Step 2:

[0415] The server performs de-noise removal on the collected data. This process filters out data that may contain noise or outliers, generating a clean dataset. For example, if a sudden spike in power consumption is temporary noise, this is corrected. The input is the raw data from step 1, and the output is the de-noise-removed data.

[0416] Step 3:

[0417] The server performs missing value imputation on the data from Step 2. This step fills in data that was intermittently unavailable both temporally and correlatively. Artificial intelligence is used to imputate incomplete data based on historical data and trends. The input is clean data with missing values, and the output is a complete dataset.

[0418] Step 4:

[0419] The server uses a machine learning model to analyze the data from step 3. Here, it learns user activity patterns and emotional states to form standard patterns. Libraries such as TensorFlow and PyTorch are used for this analysis. The input is the complete dataset, and the output is the analyzed pattern information.

[0420] Step 5:

[0421] The server performs anomaly detection based on the analyzed pattern information. It sets specific thresholds and evaluates the data against them to identify anomalies. For example, this might apply if water usage significantly exceeds normal levels. The input consists of the analyzed pattern information and current data, while the output indicates whether an anomaly exists and provides detailed information about it.

[0422] Step 6:

[0423] If an anomaly is detected, the server immediately sends a real-time notification to the user's mobile device. This notification is sent using Firebase Cloud Messaging, etc. The prompt message sent will be, "A deviation from your normal daily routine has been detected. Please check immediately." The input is the anomaly detection information, and the output is the notification sent to the user's device.

[0424] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0425] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0426] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0427] [Third Embodiment]

[0428] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0429] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0430] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0431] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0432] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0433] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0434] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0435] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0436] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0437] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0438] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0439] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0440] This invention is a system for monitoring the living conditions of elderly people. It combines machine learning technology to analyze the usage of electricity, water, and gas within the home, monitors living patterns, and detects abnormalities. The operation flow of this system is described below.

[0441] First, several types of sensors are installed in the user's home. These sensors include power consumption meters, flow sensors, and voice sensors. These sensors monitor the use of electricity, gas, and water in the home in real time.

[0442] Data obtained from each sensor is transmitted to a server via the network through a terminal. The server removes noise from the received data and appropriately imputes missing data. This preprocessing is performed to improve the accuracy of the analysis.

[0443] Next, the server inputs the pre-processed data into an artificial intelligence model for analysis. This AI model learns the user's lifestyle patterns based on past data and builds a model of normal behavior. For example, it can detect when electricity consumption increases each morning and recognize the user's wake-up pattern.

[0444] Based on established usage patterns, the server sets criteria for detecting anomalies. These include deviations from normal usage patterns and periods of absence from use. When an anomaly is detected, the server sends an alert to the user's family or caregiver's device.

[0445] As a concrete example, suppose an elderly person has a routine of using an electric kettle at 8 AM every morning, followed by a shower. If this routine suddenly stops completely, the server will determine it's an anomaly based on defined criteria and send an alert. The user's family can receive a notification on their smartphone and quickly check the situation.

[0446] Finally, based on user feedback, the server continuously refines its machine learning model to further improve the system's accuracy. This approach enables continuous monitoring without requiring elderly individuals to operate specific technological devices.

[0447] The following describes the processing flow.

[0448] Step 1:

[0449] Various sensors installed in the user's home measure the usage of electricity, water, gas, and voice, and transmit the data to a terminal.

[0450] Step 2:

[0451] The terminal converts the collected data into a format suitable for transfer to the server and sends it to the server via the network.

[0452] Step 3:

[0453] The server performs noise reduction processing and handles missing values ​​on the received raw data. Specifically, it cleans up the data using appropriate filtering techniques.

[0454] Step 4:

[0455] The server inputs pre-processed data into an artificial intelligence (AI) analysis unit, uses a trained model to analyze the user's daily life patterns, and constructs a baseline daily rhythm.

[0456] Step 5:

[0457] The server sets anomaly detection criteria based on the established lifestyle rhythm, and monitors for abnormal patterns while performing real-time data analysis.

[0458] Step 6:

[0459] When an anomaly is detected, for example, if there is no usage for an extended period during normal operating hours, the server generates an alert and prepares to send a warning to pre-registered recipients.

[0460] Step 7:

[0461] The device (for example, the user's family member or caregiver's smartphone) receives an alert notification from the server and immediately displays the notification to the user. This allows them to check the situation.

[0462] Step 8:

[0463] If a false positive occurs based on user feedback, the server receives the feedback data, adjusts the AI ​​model, and improves the accuracy of subsequent analyses.

[0464] (Example 1)

[0465] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0466] There is a need to provide a safe and secure living environment for the elderly and those requiring care, but conventional monitoring systems lack the accuracy to detect abnormal behavior in real time. Furthermore, it is difficult to conduct unbiased monitoring while protecting user privacy. It is necessary to solve these problems and realize a safe and efficient monitoring system.

[0467] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0468] In this invention, the server includes means for acquiring the usage status of energy resources in the environment via one or more sensing devices, means for performing noise reduction and missing information imputation for preprocessing the acquired information, and means for analyzing the preprocessed information using a machine learning algorithm to learn the behavioral patterns of the person being monitored. This enables high-precision monitoring of the person's lifestyle and allows for a quick and appropriate response when an anomaly is detected. Furthermore, it enables continuous monitoring while protecting privacy.

[0469] A "sensing device" is a device used to measure the usage of energy resources such as electricity, water, and gas in the environment and to acquire that information.

[0470] "Information preprocessing" is a process that improves data quality by removing noise and filling in missing information from acquired data.

[0471] A "machine learning algorithm" is a computational method that identifies patterns from large amounts of data and learns from them to make predictions and classifications about future data.

[0472] A "behavioral pattern" is a set of characteristics that indicate the regularity or tendencies of the daily activities and behaviors of the person being monitored.

[0473] An "abnormal pattern" refers to irregular activity or behavior that differs from the normal behavioral patterns learned by a machine learning algorithm.

[0474] A "criterion or threshold" is a criterion or limit used to make a judgment when detecting an unusual pattern.

[0475] A "warning signal" is an alarm message sent to a pre-configured recipient when an unusual pattern is detected.

[0476] A "mobile communication device" is a portable device that a user can carry and use to receive communications and notifications from external sources.

[0477] This invention is a monitoring system primarily for safely observing the lives of the elderly and those requiring care. The system operates as follows:

[0478] Multiple sensing devices are installed in the user's environment. These devices measure the usage of energy resources such as electricity, water, and gas in real time. This allows for detailed monitoring of usage patterns in daily life.

[0479] The collected information is transmitted to the server via the terminal. Upon arrival at the server, the information undergoes preprocessing, including noise reduction and the imputation of missing information. This preprocessing improves the accuracy of the data, enabling more reliable analysis.

[0480] The pre-processed data is then analyzed using a machine learning algorithm. The server uses this algorithm to learn the behavioral patterns of the monitored individuals. For example, it can identify patterns such as whether the monitored individual uses an electric kettle at a specific time each day.

[0481] Based on the analysis results, the server sets criteria and thresholds for detecting abnormal behavior. This process allows for rapid detection of unusual behavior.

[0482] When an anomaly is detected, the server sends an alert signal to pre-configured recipients. For example, if a monitored person stops using an electrical appliance they normally use, an alert can be quickly sent to their family. This alert signal is delivered as a notification to the user's mobile device, allowing for prompt action.

[0483] Furthermore, by utilizing feedback from users and their families, the server continuously adjusts the parameters of its machine learning algorithms to improve the system's accuracy. This dynamic adjustment allows the system to flexibly adapt to changing environments and new behavioral patterns.

[0484] As described above, this system uses a generative AI model to achieve effective monitoring. For example, by inputting prompts such as, "How can we improve the way we learn the lifestyle patterns of elderly people and notify them of abnormalities?" into the generative AI model, it is possible to find even more efficient solutions.

[0485] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0486] Step 1:

[0487] Users install multiple sensing devices in their homes. These devices collect real-time data on electricity, water, and gas usage. The input is data on the use of various energy resources, and the output is time-series usage patterns of the surrounding environment. This process enables detailed monitoring of the residents' lives.

[0488] Step 2:

[0489] The terminal transmits the collected data to the server via the network. The input is raw data from the sensing device, and the output is secure data packets that reach the server. Here, data encryption technology is used to ensure the security of the information and prevent unauthorized access from external sources.

[0490] Step 3:

[0491] The server preprocesses the received data. Specifically, it applies a denoising filter and imputes missing data based on statistical methods. The input is raw data sent from the terminal, and the output is a reliable and clean dataset. This process improves analytical performance.

[0492] Step 4:

[0493] The server inputs pre-processed data into a machine learning model and performs the analysis. The algorithm repeatedly learns from the data and extracts the user's typical behavioral patterns. The input is a clean dataset, and the output generates the behavioral patterns of the monitored individual. This analysis identifies daily habits.

[0494] Step 5:

[0495] The server sets anomaly detection criteria based on the analysis results. A threshold is set, and any behavior exceeding it is detected as an anomaly. The input is behavioral pattern data, and the output is the formulation of anomaly detection rules. This enables the immediate detection of behavior outside the normal range.

[0496] Step 6:

[0497] If an anomaly is detected, the server sends a warning signal. An alarm is sent to the terminal of a pre-configured recipient. The input is the result of the anomaly detection algorithm, and the output is an alert notification sent to the recipient. This notification prompts a quick response.

[0498] Step 7:

[0499] The user or their representative sends feedback to the server. The server uses this feedback to adjust the parameters of the machine learning model, improving the system's accuracy. The input is the feedback information, and the output is the optimized model parameters. This adjustment leads to continuous improvement.

[0500] (Application Example 1)

[0501] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0502] There is a need to efficiently and continuously detect abnormalities within the home and respond quickly while ensuring the safety of the elderly. Conventional systems only monitor electricity and water usage, making it difficult to immediately detect unauthorized access from external sources or suspicious usage patterns. This invention aims to solve these problems and enhance the safety of the home and its surrounding environment.

[0503] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0504] In this invention, the server includes means for acquiring energy and liquid usage within the home via one or more detection means; means for noise reduction and missing value processing for preprocessing the acquired information; means for analyzing the preprocessed information using a machine learning model to learn user behavior patterns; and means for detecting unauthorized use from external sources, recording the anomaly, and notifying the user. This enables early detection of potential dangers faced by the elderly within the home and allows for a rapid response.

[0505] "Detection means" refers to a device or group of devices installed to monitor energy and liquid usage within a household.

[0506] "Preprocessing" is the process of removing noise from acquired information and filling in missing values.

[0507] A "machine learning model" is an artificial intelligence-based algorithm used to learn and analyze user behavior patterns from pre-processed information.

[0508] A "rule or criterion" is a rule or threshold set to detect anomaly patterns based on the analyzed information.

[0509] "Communication means" refers to a means of sending an alarm to a pre-configured recipient when an abnormal pattern is detected.

[0510] "Evaluation" refers to opinions and information regarding the operation of a system received as feedback from users.

[0511] "Conditions" refer to the parameters or settings of a machine learning model that are changed when tuning the model.

[0512] "Detecting unauthorized use" refers to the process of discovering unauthorized use or access from outside the home or its surroundings.

[0513] "Energy and liquids" refers to infrastructure resources such as electricity and water consumed within the home.

[0514] The system for implementing this invention has the function of monitoring and analyzing household electricity and water usage to detect anomalies in order to ensure the safety of the elderly. This system is operated using a server, terminals, a group of sensors, and a machine learning model.

[0515] The server receives data acquired from multiple sensors installed within the home, including power consumption meters and flow sensors. The received data undergoes noise reduction and interpolation on the server, and is pre-processed for analysis.

[0516] Next, the server uses the pre-processed data to perform analysis with an AI model that employs machine learning libraries such as TensorFlow. This AI model has already learned the normal lifestyle patterns of elderly people from past data and sets criteria for detecting anomalies based on this.

[0517] If an anomaly is detected, the server uses Firebase to send a notification to the user's mobile device. This allows family members and caregivers to immediately check the situation and take necessary action.

[0518] This system also periodically adjusts the parameters of the AI ​​model based on evaluations and feedback from users. This ensures that the accuracy of the analysis is continuously improved.

[0519] For example, if water usage occurs at an unusual time during a family trip, the system will immediately detect this anomaly and send an alert to a mobile device. The user can then receive the notification and check the safety of their home.

[0520] Examples of prompt statements are as follows:

[0521] "Detect any anomalies in the electricity and water usage pattern data for the past 7 days. If there are deviations from normal conditions, analyze the reasons and create a report."

[0522] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0523] Step 1:

[0524] Multiple sensors installed in the home collect electricity and water usage data in real time. The system receives data from each sensor as input. This includes electricity meters, flow sensors, and acoustic sensors. The collected data is immediately transmitted to a server.

[0525] Step 2:

[0526] The server performs noise reduction and imputation on the received sensor data. The input is raw data from the sensor, and the output is a clean dataset with noise removed and missing values ​​imputed. Specifically, it filters out inconsistent data points and applies imputation to maintain the continuity of the time-series data.

[0527] Step 3:

[0528] The server inputs preprocessed data into a machine learning model using TensorFlow and performs analysis. The input is a clean dataset, and the output is the user's everyday behavior patterns. The machine learning model has already learned patterns based on past data and generates a behavioral model to build criteria for anomaly detection.

[0529] Step 4:

[0530] Based on the analysis results, the server sets thresholds for detecting anomalies and continues monitoring. The input is the behavioral patterns from the machine learning model, and the output is the threshold for anomaly detection. The server continuously evaluates whether there are any signs of anomaly.

[0531] Step 5:

[0532] When an anomaly is detected, the server uses Firebase to instruct the system to send an alert to the user's mobile device. The input is the result of the anomaly detection, and the output is the alert notification. The server sends an emergency alert to the user's device, prompting immediate action on the problem.

[0533] Step 6:

[0534] The user or their family member will check the details of the anomaly through the terminal and take appropriate action as needed. The input is an alarm notification, and the output is the user's verification of the anomaly and the corresponding actions taken. Based on the received notification, the user will check the safety of the elderly person and, if necessary, contact other relevant parties.

[0535] Step 7:

[0536] User feedback is collected, and the server uses this feedback to adjust the machine learning model. The input is the user feedback, and the output is the parameters of the adjusted model. This allows the system to improve the accuracy of analysis and anomaly detection.

[0537] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0538] This invention provides a system that enables monitoring of household electricity, water, and gas usage, as well as emotional states based on sound and video. It analyzes this data to monitor the user's lifestyle and emotional patterns. The operation of this system is described in detail below.

[0539] First, the various sensors installed in the user's home include power consumption meters, flow sensors, sound sensors, and cameras that acquire visual data. These sensors collect data on daily life within the home, and the data is transmitted to a server via a terminal.

[0540] The server performs noise reduction and data imputation on electricity, water, and gas data to generate standardized datasets. Acoustic and video data undergo similar preprocessing. This results in clean and consistent data.

[0541] Next, the server utilizes an artificial intelligence (AI) analysis module to analyze lifestyle patterns and emotional states based on this data. In analyzing lifestyle patterns, it learns a standard behavioral timetable based on conventional electricity and water usage and detects anomalies. The emotion engine analyzes information obtained from audio and video to recognize emotions from the user's facial expressions and tone of voice, and stores these as daily emotional patterns.

[0542] Using the data collected in this way, the server can monitor not only abnormal lifestyle patterns but also abnormal emotional states. If an anomaly is detected, the server sends an alert to a pre-designated recipient, such as a family member or caregiver. Specifically, if usage patterns deviate significantly from normal lifestyle patterns or if it is determined that the user is experiencing high levels of emotional stress, the server will promptly notify the user accordingly.

[0543] Furthermore, feedback from users and recipients is collected, and the AI ​​model is adjusted to improve the accuracy of the analysis. This feedback loop allows the system to be continuously optimized, providing a natural monitoring environment that does not require elderly people to perform specific actions.

[0544] The following describes the processing flow.

[0545] Step 1:

[0546] Various sensors and cameras installed in the user's home are operating, continuously measuring electricity, water, and gas usage, as well as acoustic and visual data. The data is collected on a terminal in real time.

[0547] Step 2:

[0548] The terminal formats the collected data and sends it to the server via the communication line. This process minimizes data loss and enables smooth data transmission and reception.

[0549] Step 3:

[0550] The server denoises the received data and fills in any missing data based on past data. At this stage, consistency processing is performed to improve the reliability of the data.

[0551] Step 4:

[0552] The server inputs pre-processed data into an artificial intelligence model to analyze the user's daily life patterns and emotional patterns. Numerical data such as the frequency of electricity, water, and gas usage is plotted on a timeline to learn the user's lifestyle habits. Acoustic and visual data are used to determine the user's emotional state using facial recognition and voice analysis, and this is compared to normal conditions.

[0553] Step 5:

[0554] The server performs an anomaly monitoring process based on the lifestyle and emotional patterns obtained from the analysis. Here, deviations from the normal are determined based on defined thresholds and rules. For example, if usage frequency drops significantly or emotional stress is assessed as high, it will be treated as an anomaly.

[0555] Step 6:

[0556] If an anomaly is detected, the server generates an alarm and sends a notification to pre-registered family members or caregivers via the terminal. The notification includes the detected anomaly and its details, allowing recipients to consider immediate action.

[0557] Step 7:

[0558] Users or recipients report false positives and areas for system improvement through an interface designed to collect feedback. This feedback is recorded by the server and used to readjust the AI ​​model.

[0559] Step 8:

[0560] Based on feedback, the server optimizes the parameters of the artificial intelligence model, improving the accuracy of anomaly detection and alert transmission. This allows the system to continuously evolve and support a safe and secure living environment for the elderly.

[0561] (Example 2)

[0562] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0563] Conventional home monitoring systems only track electricity, water, and gas usage, making it difficult to monitor users' lifestyles and emotional states in detail. Furthermore, they lacked the ability to detect anomalies early and provide appropriate feedback, making it challenging to effectively support users' health and safety. Therefore, there is a need for a system that comprehensively analyzes overall household data and provides rapid and appropriate notification of anomalies.

[0564] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0565] In this invention, the server includes means for acquiring indoor resource consumption and environmental data, means for applying removal filters and complementary algorithms, and means for analyzing the data using an intelligent analysis engine and learning individual behavioral patterns. This makes it possible to analyze lifestyle patterns and emotional states with high accuracy, quickly detect anomalies, and send warnings.

[0566] A "detection device" refers to an instrument or sensor used to measure the state of the environment or an object, and includes, for example, power consumption meters, flow sensors, sound detection sensors, and cameras.

[0567] "Resource consumption" refers to the consumption of energy and resources such as electricity, water, and gas used indoors.

[0568] "Environmental data" refers to information such as sound and video in a specific space, including acoustic and visual information.

[0569] A "removal filter" refers to an algorithm used to remove unnecessary information, such as noise, from data.

[0570] A "complementary algorithm" refers to a method for inferring or reconstructing missing parts of collected data.

[0571] An "intelligent analysis engine" refers to artificial intelligence technology that highly analyzes collected data and identifies patterns.

[0572] "Behavioral patterns" refer to a timeline based on past data that shows an individual's lifestyle habits and tendencies in daily behavior.

[0573] The system for realizing this invention enables detailed monitoring of resource consumption and environmental data within the home. The main hardware consists of multiple sensor groups and a data processing unit. Specifically, it includes a power consumption meter, flow sensors, sound sensors, and cameras. These sensors are responsible for acquiring data on electricity, water, and gas consumption within the home, as well as audio and video data.

[0574] The user sends the collected data to the server via their device. The device acts as a temporary relay point for the data, ensuring communication stability. After transmission, the data is received by the server and preprocessed using denoising filters and imputation algorithms. This process removes noise and infers and imputes missing data, resulting in a clean and consistent dataset.

[0575] Next, the server starts up the intelligent analysis engine and analyzes the pre-processed data. Generative AI models are used for the analysis, which are capable of precisely learning individual behavioral patterns and emotional states. For example, this includes methods for learning normal behavioral patterns from power usage data, and methods for recognizing emotions from voice tone and facial expressions. Anomaly detection is then performed using the data analyzed in this way.

[0576] If an anomaly is detected, the server immediately sends an alert to the terminals of pre-configured recipients (e.g., family members or medical personnel). This is a crucial function for continuously monitoring the user's health and safety. Furthermore, feedback from users and recipients is collected and used to refine the accuracy of the intelligent analysis engine. This process automatically optimizes the system, enabling more effective monitoring.

[0577] A concrete example of a prompt message could be, "Explain a mechanism that detects anomalies from a user's daily life data and notifies the user of their emotional state when stress levels are high." By entering this prompt message into the system, it becomes possible to discuss and deepen understanding of various aspects of the user's life.

[0578] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0579] Step 1:

[0580] Within the user's home, various sensors acquire power consumption, flow rate, audio, and video data in real time. This creates a detailed dataset of the user's daily life. The input is raw data transmitted from the sensors, and the output is data obtained in a format that is stored in temporary data storage.

[0581] Step 2:

[0582] The terminal receives data collected from the sensor and performs an initial integrity check. It identifies missing or inconsistent data and makes any necessary adjustments to ensure data reliability. Raw data from the sensor is used as input, and pre-processed data is obtained as output.

[0583] Step 3:

[0584] The server receives data sent from the terminal and applies denoising filters and interpolation algorithms. By reducing noisy data and imputing missing parts, it generates a clean and complete dataset. The input is the initial data sent from the terminal, and the output is the denoised and interpolated data.

[0585] Step 4:

[0586] The server analyzes pre-processed data using a generation AI model. It analyzes power usage patterns, voice tone, and facial expression data to learn lifestyle patterns and emotional states. The input is a clean dataset, and the output generates a model of the user's behavioral patterns and emotional states.

[0587] Step 5:

[0588] The server performs anomaly detection based on the generated model. It compares the analysis results with pre-set criteria and thresholds to identify behaviors and emotional states that deviate from normal patterns. Behavioral patterns and emotional state models are used as input, and the anomaly determination result is obtained as output.

[0589] Step 6:

[0590] The server sends an alert to pre-configured recipients if an anomaly is detected. The recipients' devices receive notifications, providing alerts regarding safety and health management. The input is the anomaly detection result, and the output is the notification sent to the recipient's device.

[0591] Step 7:

[0592] Users and recipients provide feedback to the system. Based on this feedback, the generated AI model is recalibrated, improving the accuracy of the analysis. Feedback information is used as input, and the improved model is obtained as output.

[0593] (Application Example 2)

[0594] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0595] In systems aimed at improving safety and security in daily life by comprehensively monitoring energy, resource, and water consumption within the home, as well as the emotional state of residents, and quickly detecting and notifying users of anomalies, conventional systems have the challenge of handling individual data separately, making unified monitoring difficult. Furthermore, there are concerns that delays in anomaly detection could lead to inadequate responses in situations requiring rapid action. In addition, there is a need to detect changes in emotional state early and enable appropriate feedback.

[0596] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0597] In this invention, the server includes means for acquiring energy, water, and fuel usage in a living space via one or more detection devices; means for preprocessing the acquired data by removing unwanted components and imputing missing values; and means for analyzing the preprocessed data using a machine learning model to learn the user's activity patterns. This makes it possible to detect abnormalities in the resident's emotional state and activity patterns in real time and promptly notify the resident's mobile communication terminal.

[0598] A "detection device" is a device used to measure the usage of electricity, water, and fuel within a living space. This includes sensors and meters.

[0599] "Removal of unwanted components" is a data preprocessing operation that removes noise and outliers contained within the data.

[0600] "Missing value imputation" is the operation of filling in missing values ​​in an incomplete dataset to generate complete data suitable for analysis.

[0601] A "machine learning model" is a set of algorithms used to learn patterns from data and perform analysis and predictions.

[0602] "Activity patterns" refer to a series of actions and tendencies related to a user's energy and resource consumption.

[0603] An "abnormality" is a sign that indicates behavior or a situation that deviates from normal activity patterns, and often requires a prompt response.

[0604] "Real-time notification" is a process that immediately transmits information about analyzed anomalies and emotional states to the user's mobile communication terminal.

[0605] The system implementing this invention centrally monitors the consumption of electricity, water, and fuel within the home, analyzes the emotional state of the residents to detect anomalies, and provides prompt notification. The server operates this system in the following specific manner.

[0606] First, the server collects energy and resource usage data from multiple detection devices. These include electricity meters, flow sensors, and gas detectors. These devices record specific consumption patterns within the living space in real time and transmit the data to the server.

[0607] Next, the server removes unwanted components and imputes missing values ​​from the received data to generate a clean dataset suitable for analysis. Specifically, it processes noise and anomalous missing data, transforming it into complete and consistent data.

[0608] Subsequently, the server operates a machine learning model based on the prepared data to learn the activity patterns of the residents. This model typically uses open-source libraries such as TensorFlow or PyTorch. The machine learning model analyzes past consumption data and acoustic information to detect stable behavioral patterns and emotional changes.

[0609] If an anomaly is detected, for example, if water usage increases significantly more than usual and acoustic data indicates feelings of anxiety, the server immediately activates a real-time notification function. This allows the user's mobile device to be notified of the anomaly. This notification is sent via a communication platform such as Firebase Cloud Messaging.

[0610] For example, if water leakage continues for an extended period, or if stress or anxiety is detected from the user's voice, a prompt message such as "A deviation from your normal lifestyle has been detected. Please check immediately." will be generated and sent as a notification.

[0611] In this way, the server functions as a system that enhances the safety and sense of security in the living space.

[0612] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0613] Step 1:

[0614] The server collects data from various detection devices installed in the home. Specifically, it acquires data in real time from devices such as power meters, flow sensors, and gas meters, and transmits it to the server. The input is raw data from these devices, and the output is unprocessed data aggregated on the server.

[0615] Step 2:

[0616] The server performs de-noise removal on the collected data. This process filters out data that may contain noise or outliers, generating a clean dataset. For example, if a sudden spike in power consumption is temporary noise, this is corrected. The input is the raw data from step 1, and the output is the de-noise-removed data.

[0617] Step 3:

[0618] The server performs missing value imputation on the data from Step 2. This step fills in data that was intermittently unavailable both temporally and correlatively. Artificial intelligence is used to imputate incomplete data based on historical data and trends. The input is clean data with missing values, and the output is a complete dataset.

[0619] Step 4:

[0620] The server uses a machine learning model to analyze the data from step 3. Here, it learns user activity patterns and emotional states to form standard patterns. Libraries such as TensorFlow and PyTorch are used for this analysis. The input is the complete dataset, and the output is the analyzed pattern information.

[0621] Step 5:

[0622] The server performs anomaly detection based on the analyzed pattern information. It sets specific thresholds and evaluates the data against them to identify anomalies. For example, this might apply if water usage significantly exceeds normal levels. The input consists of the analyzed pattern information and current data, while the output indicates whether an anomaly exists and provides detailed information about it.

[0623] Step 6:

[0624] If an anomaly is detected, the server immediately sends a real-time notification to the user's mobile device. This notification is sent using Firebase Cloud Messaging, etc. The prompt message sent will be, "A deviation from your normal daily routine has been detected. Please check immediately." The input is the anomaly detection information, and the output is the notification sent to the user's device.

[0625] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0626] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0627] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0628] [Fourth Embodiment]

[0629] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0630] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0631] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0632] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0633] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0635] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0636] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0637] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0638] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0639] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0640] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0641] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0642] This invention is a system for monitoring the living conditions of elderly people. It combines machine learning technology to analyze the usage of electricity, water, and gas within the home, monitors living patterns, and detects abnormalities. The operation flow of this system is described below.

[0643] First, several types of sensors are installed in the user's home. These sensors include power consumption meters, flow sensors, and voice sensors. These sensors monitor the use of electricity, gas, and water in the home in real time.

[0644] Data obtained from each sensor is transmitted to a server via the network through a terminal. The server removes noise from the received data and appropriately imputes missing data. This preprocessing is performed to improve the accuracy of the analysis.

[0645] Next, the server inputs the pre-processed data into an artificial intelligence model for analysis. This AI model learns the user's lifestyle patterns based on past data and builds a model of normal behavior. For example, it can detect when electricity consumption increases each morning and recognize the user's wake-up pattern.

[0646] Based on established usage patterns, the server sets criteria for detecting anomalies. These include deviations from normal usage patterns and periods of absence from use. When an anomaly is detected, the server sends an alert to the user's family or caregiver's device.

[0647] As a concrete example, suppose an elderly person has a routine of using an electric kettle at 8 AM every morning, followed by a shower. If this routine suddenly stops completely, the server will determine it's an anomaly based on defined criteria and send an alert. The user's family can receive a notification on their smartphone and quickly check the situation.

[0648] Finally, based on user feedback, the server continuously refines its machine learning model to further improve the system's accuracy. This approach enables continuous monitoring without requiring elderly individuals to operate specific technological devices.

[0649] The following describes the processing flow.

[0650] Step 1:

[0651] Various sensors installed in the user's home measure the usage of electricity, water, gas, and voice, and transmit the data to a terminal.

[0652] Step 2:

[0653] The terminal converts the collected data into a format suitable for transfer to the server and sends it to the server via the network.

[0654] Step 3:

[0655] The server performs noise reduction processing and handles missing values ​​on the received raw data. Specifically, it cleans up the data using appropriate filtering techniques.

[0656] Step 4:

[0657] The server inputs pre-processed data into an artificial intelligence (AI) analysis unit, uses a trained model to analyze the user's daily life patterns, and constructs a baseline daily rhythm.

[0658] Step 5:

[0659] The server sets anomaly detection criteria based on the established lifestyle rhythm, and monitors for abnormal patterns while performing real-time data analysis.

[0660] Step 6:

[0661] When an anomaly is detected, for example, if there is no usage for an extended period during normal operating hours, the server generates an alert and prepares to send a warning to pre-registered recipients.

[0662] Step 7:

[0663] The device (for example, the user's family member or caregiver's smartphone) receives an alert notification from the server and immediately displays the notification to the user. This allows them to check the situation.

[0664] Step 8:

[0665] If a false positive occurs based on user feedback, the server receives the feedback data, adjusts the AI ​​model, and improves the accuracy of subsequent analyses.

[0666] (Example 1)

[0667] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] There is a need to provide a safe and secure living environment for the elderly and those requiring care, but conventional monitoring systems lack the accuracy to detect abnormal behavior in real time. Furthermore, it is difficult to conduct unbiased monitoring while protecting user privacy. It is necessary to solve these problems and realize a safe and efficient monitoring system.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0670] In this invention, the server includes means for acquiring the usage status of energy resources in the environment via one or more sensing devices, means for performing noise reduction and missing information imputation for preprocessing the acquired information, and means for analyzing the preprocessed information using a machine learning algorithm to learn the behavioral patterns of the person being monitored. This enables high-precision monitoring of the person's lifestyle and allows for a quick and appropriate response when an anomaly is detected. Furthermore, it enables continuous monitoring while protecting privacy.

[0671] A "sensing device" is a device used to measure the usage of energy resources such as electricity, water, and gas in the environment and to acquire that information.

[0672] "Information preprocessing" is a process that improves data quality by removing noise and filling in missing information from acquired data.

[0673] A "machine learning algorithm" is a computational method that identifies patterns from large amounts of data and learns from them to make predictions and classifications about future data.

[0674] A "behavioral pattern" is a set of characteristics that indicate the regularity or tendencies of the daily activities and behaviors of the person being monitored.

[0675] An "abnormal pattern" refers to irregular activity or behavior that differs from the normal behavioral patterns learned by a machine learning algorithm.

[0676] A "criterion or threshold" is a criterion or limit used to make a judgment when detecting an unusual pattern.

[0677] A "warning signal" is an alarm message sent to a pre-configured recipient when an unusual pattern is detected.

[0678] A "mobile communication device" is a portable device that a user can carry and use to receive communications and notifications from external sources.

[0679] This invention is a monitoring system primarily for safely observing the lives of the elderly and those requiring care. The system operates as follows:

[0680] Multiple sensing devices are installed in the user's environment. These devices measure the usage of energy resources such as electricity, water, and gas in real time. This allows for detailed monitoring of usage patterns in daily life.

[0681] The collected information is transmitted to the server via the terminal. Upon arrival at the server, the information undergoes preprocessing, including noise reduction and the imputation of missing information. This preprocessing improves the accuracy of the data, enabling more reliable analysis.

[0682] The pre-processed data is then analyzed using a machine learning algorithm. The server uses this algorithm to learn the behavioral patterns of the monitored individuals. For example, it can identify patterns such as whether the monitored individual uses an electric kettle at a specific time each day.

[0683] Based on the analysis results, the server sets criteria and thresholds for detecting abnormal behavior. This process allows for rapid detection of unusual behavior.

[0684] When an anomaly is detected, the server sends an alert signal to pre-configured recipients. For example, if a monitored person stops using an electrical appliance they normally use, an alert can be quickly sent to their family. This alert signal is delivered as a notification to the user's mobile device, allowing for prompt action.

[0685] Furthermore, by utilizing feedback from users and their families, the server continuously adjusts the parameters of its machine learning algorithms to improve the system's accuracy. This dynamic adjustment allows the system to flexibly adapt to changing environments and new behavioral patterns.

[0686] As described above, this system uses a generative AI model to achieve effective monitoring. For example, by inputting prompts such as, "How can we improve the way we learn the lifestyle patterns of elderly people and notify them of abnormalities?" into the generative AI model, it is possible to find even more efficient solutions.

[0687] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0688] Step 1:

[0689] Users install multiple sensing devices in their homes. These devices collect real-time data on electricity, water, and gas usage. The input is data on the use of various energy resources, and the output is time-series usage patterns of the surrounding environment. This process enables detailed monitoring of the residents' lives.

[0690] Step 2:

[0691] The terminal transmits the collected data to the server via the network. The input is raw data from the sensing device, and the output is secure data packets that reach the server. Here, data encryption technology is used to ensure the security of the information and prevent unauthorized access from external sources.

[0692] Step 3:

[0693] The server preprocesses the received data. Specifically, it applies a denoising filter and imputes missing data based on statistical methods. The input is raw data sent from the terminal, and the output is a reliable and clean dataset. This process improves analytical performance.

[0694] Step 4:

[0695] The server inputs pre-processed data into a machine learning model and performs the analysis. The algorithm repeatedly learns from the data and extracts the user's typical behavioral patterns. The input is a clean dataset, and the output generates the behavioral patterns of the monitored individual. This analysis identifies daily habits.

[0696] Step 5:

[0697] The server sets anomaly detection criteria based on the analysis results. A threshold is set, and any behavior exceeding it is detected as an anomaly. The input is behavioral pattern data, and the output is the formulation of anomaly detection rules. This enables the immediate detection of behavior outside the normal range.

[0698] Step 6:

[0699] If an anomaly is detected, the server sends a warning signal. An alarm is sent to the terminal of a pre-configured recipient. The input is the result of the anomaly detection algorithm, and the output is an alert notification sent to the recipient. This notification prompts a quick response.

[0700] Step 7:

[0701] The user or their representative sends feedback to the server. The server uses this feedback to adjust the parameters of the machine learning model, improving the system's accuracy. The input is the feedback information, and the output is the optimized model parameters. This adjustment leads to continuous improvement.

[0702] (Application Example 1)

[0703] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0704] There is a need to efficiently and continuously detect abnormalities within the home and respond quickly while ensuring the safety of the elderly. Conventional systems only monitor electricity and water usage, making it difficult to immediately detect unauthorized access from external sources or suspicious usage patterns. This invention aims to solve these problems and enhance the safety of the home and its surrounding environment.

[0705] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0706] In this invention, the server includes means for acquiring energy and liquid usage within the home via one or more detection means; means for noise reduction and missing value processing for preprocessing the acquired information; means for analyzing the preprocessed information using a machine learning model to learn user behavior patterns; and means for detecting unauthorized use from external sources, recording the anomaly, and notifying the user. This enables early detection of potential dangers faced by the elderly within the home and allows for a rapid response.

[0707] "Detection means" refers to a device or group of devices installed to monitor energy and liquid usage within a household.

[0708] "Preprocessing" is the process of removing noise from acquired information and filling in missing values.

[0709] A "machine learning model" is an artificial intelligence-based algorithm used to learn and analyze user behavior patterns from pre-processed information.

[0710] A "rule or criterion" is a rule or threshold set to detect anomaly patterns based on the analyzed information.

[0711] "Communication means" refers to a means of sending an alarm to a pre-configured recipient when an abnormal pattern is detected.

[0712] "Evaluation" refers to opinions and information regarding the operation of a system received as feedback from users.

[0713] "Conditions" refer to the parameters or settings of a machine learning model that are changed when tuning the model.

[0714] "Detecting unauthorized use" refers to the process of discovering unauthorized use or access from outside the home or its surroundings.

[0715] "Energy and liquids" refers to infrastructure resources such as electricity and water consumed within the home.

[0716] The system for implementing this invention has the function of monitoring and analyzing household electricity and water usage to detect anomalies in order to ensure the safety of the elderly. This system is operated using a server, terminals, a group of sensors, and a machine learning model.

[0717] The server receives data acquired from multiple sensors installed within the home, including power consumption meters and flow sensors. The received data undergoes noise reduction and interpolation on the server, and is pre-processed for analysis.

[0718] Next, the server uses the pre-processed data to perform analysis with an AI model that employs machine learning libraries such as TensorFlow. This AI model has already learned the normal lifestyle patterns of elderly people from past data and sets criteria for detecting anomalies based on this.

[0719] If an anomaly is detected, the server uses Firebase to send a notification to the user's mobile device. This allows family members and caregivers to immediately check the situation and take necessary action.

[0720] This system also periodically adjusts the parameters of the AI ​​model based on evaluations and feedback from users. This ensures that the accuracy of the analysis is continuously improved.

[0721] For example, if water usage occurs at an unusual time during a family trip, the system will immediately detect this anomaly and send an alert to a mobile device. The user can then receive the notification and check the safety of their home.

[0722] Examples of prompt statements are as follows:

[0723] "Detect any anomalies in the electricity and water usage pattern data for the past 7 days. If there are deviations from normal conditions, analyze the reasons and create a report."

[0724] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0725] Step 1:

[0726] Multiple sensors installed in the home collect electricity and water usage data in real time. The system receives data from each sensor as input. This includes electricity meters, flow sensors, and acoustic sensors. The collected data is immediately transmitted to a server.

[0727] Step 2:

[0728] The server performs noise reduction and imputation on the received sensor data. The input is raw data from the sensor, and the output is a clean dataset with noise removed and missing values ​​imputed. Specifically, it filters out inconsistent data points and applies imputation to maintain the continuity of the time-series data.

[0729] Step 3:

[0730] The server inputs preprocessed data into a machine learning model using TensorFlow and performs analysis. The input is a clean dataset, and the output is the user's everyday behavior patterns. The machine learning model has already learned patterns based on past data and generates a behavioral model to build criteria for anomaly detection.

[0731] Step 4:

[0732] Based on the analysis results, the server sets thresholds for detecting anomalies and continues monitoring. The input is the behavioral patterns from the machine learning model, and the output is the threshold for anomaly detection. The server continuously evaluates whether there are any signs of anomaly.

[0733] Step 5:

[0734] When an anomaly is detected, the server uses Firebase to instruct the system to send an alert to the user's mobile device. The input is the result of the anomaly detection, and the output is the alert notification. The server sends an emergency alert to the user's device, prompting immediate action on the problem.

[0735] Step 6:

[0736] The user or their family member will check the details of the anomaly through the terminal and take appropriate action as needed. The input is an alarm notification, and the output is the user's verification of the anomaly and the corresponding actions taken. Based on the received notification, the user will check the safety of the elderly person and, if necessary, contact other relevant parties.

[0737] Step 7:

[0738] User feedback is collected, and the server uses this feedback to adjust the machine learning model. The input is the user feedback, and the output is the parameters of the adjusted model. This allows the system to improve the accuracy of analysis and anomaly detection.

[0739] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0740] This invention provides a system that enables monitoring of household electricity, water, and gas usage, as well as emotional states based on sound and video. It analyzes this data to monitor the user's lifestyle and emotional patterns. The operation of this system is described in detail below.

[0741] First, the various sensors installed in the user's home include power consumption meters, flow sensors, sound sensors, and cameras that acquire visual data. These sensors collect data on daily life within the home, and the data is transmitted to a server via a terminal.

[0742] The server performs noise reduction and data imputation on electricity, water, and gas data to generate standardized datasets. Acoustic and video data undergo similar preprocessing. This results in clean and consistent data.

[0743] Next, the server utilizes an artificial intelligence (AI) analysis module to analyze lifestyle patterns and emotional states based on this data. In analyzing lifestyle patterns, it learns a standard behavioral timetable based on conventional electricity and water usage and detects anomalies. The emotion engine analyzes information obtained from audio and video to recognize emotions from the user's facial expressions and tone of voice, and stores these as daily emotional patterns.

[0744] Using the data collected in this way, the server can monitor not only abnormal lifestyle patterns but also abnormal emotional states. If an anomaly is detected, the server sends an alert to a pre-designated recipient, such as a family member or caregiver. Specifically, if usage patterns deviate significantly from normal lifestyle patterns or if it is determined that the user is experiencing high levels of emotional stress, the server will promptly notify the user accordingly.

[0745] Furthermore, feedback from users and recipients is collected, and the AI ​​model is adjusted to improve the accuracy of the analysis. This feedback loop allows the system to be continuously optimized, providing a natural monitoring environment that does not require elderly people to perform specific actions.

[0746] The following describes the processing flow.

[0747] Step 1:

[0748] Various sensors and cameras installed in the user's home are operating, continuously measuring electricity, water, and gas usage, as well as acoustic and visual data. The data is collected on a terminal in real time.

[0749] Step 2:

[0750] The terminal formats the collected data and sends it to the server via the communication line. This process minimizes data loss and enables smooth data transmission and reception.

[0751] Step 3:

[0752] The server denoises the received data and fills in any missing data based on past data. At this stage, consistency processing is performed to improve the reliability of the data.

[0753] Step 4:

[0754] The server inputs pre-processed data into an artificial intelligence model to analyze the user's daily life patterns and emotional patterns. Numerical data such as the frequency of electricity, water, and gas usage is plotted on a timeline to learn the user's lifestyle habits. Acoustic and visual data are used to determine the user's emotional state using facial recognition and voice analysis, and this is compared to normal conditions.

[0755] Step 5:

[0756] The server performs an anomaly monitoring process based on the lifestyle and emotional patterns obtained from the analysis. Here, deviations from the normal are determined based on defined thresholds and rules. For example, if usage frequency drops significantly or emotional stress is assessed as high, it will be treated as an anomaly.

[0757] Step 6:

[0758] If an anomaly is detected, the server generates an alarm and sends a notification to pre-registered family members or caregivers via the terminal. The notification includes the detected anomaly and its details, allowing recipients to consider immediate action.

[0759] Step 7:

[0760] Users or recipients report false positives and areas for system improvement through an interface designed to collect feedback. This feedback is recorded by the server and used to readjust the AI ​​model.

[0761] Step 8:

[0762] Based on feedback, the server optimizes the parameters of the artificial intelligence model, improving the accuracy of anomaly detection and alert transmission. This allows the system to continuously evolve and support a safe and secure living environment for the elderly.

[0763] (Example 2)

[0764] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0765] Conventional home monitoring systems only track electricity, water, and gas usage, making it difficult to monitor users' lifestyles and emotional states in detail. Furthermore, they lacked the ability to detect anomalies early and provide appropriate feedback, making it challenging to effectively support users' health and safety. Therefore, there is a need for a system that comprehensively analyzes overall household data and provides rapid and appropriate notification of anomalies.

[0766] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0767] In this invention, the server includes means for acquiring indoor resource consumption and environmental data, means for applying removal filters and complementary algorithms, and means for analyzing the data using an intelligent analysis engine and learning individual behavioral patterns. This makes it possible to analyze lifestyle patterns and emotional states with high accuracy, quickly detect anomalies, and send warnings.

[0768] A "detection device" refers to an instrument or sensor used to measure the state of the environment or an object, and includes, for example, power consumption meters, flow sensors, sound detection sensors, and cameras.

[0769] "Resource consumption" refers to the consumption of energy and resources such as electricity, water, and gas used indoors.

[0770] "Environmental data" refers to information such as sound and video in a specific space, including acoustic and visual information.

[0771] A "removal filter" refers to an algorithm used to remove unnecessary information, such as noise, from data.

[0772] A "complementary algorithm" refers to a method for inferring or reconstructing missing parts of collected data.

[0773] An "intelligent analysis engine" refers to artificial intelligence technology that highly analyzes collected data and identifies patterns.

[0774] "Behavioral patterns" refer to a timeline based on past data that shows an individual's lifestyle habits and tendencies in daily behavior.

[0775] The system for realizing this invention enables detailed monitoring of resource consumption and environmental data within the home. The main hardware consists of multiple sensor groups and a data processing unit. Specifically, it includes a power consumption meter, flow sensors, sound sensors, and cameras. These sensors are responsible for acquiring data on electricity, water, and gas consumption within the home, as well as audio and video data.

[0776] The user sends the collected data to the server via their device. The device acts as a temporary relay point for the data, ensuring communication stability. After transmission, the data is received by the server and preprocessed using denoising filters and imputation algorithms. This process removes noise and infers and imputes missing data, resulting in a clean and consistent dataset.

[0777] Next, the server starts up the intelligent analysis engine and analyzes the pre-processed data. Generative AI models are used for the analysis, which are capable of precisely learning individual behavioral patterns and emotional states. For example, this includes methods for learning normal behavioral patterns from power usage data, and methods for recognizing emotions from voice tone and facial expressions. Anomaly detection is then performed using the data analyzed in this way.

[0778] If an anomaly is detected, the server immediately sends an alert to the terminals of pre-configured recipients (e.g., family members or medical personnel). This is a crucial function for continuously monitoring the user's health and safety. Furthermore, feedback from users and recipients is collected and used to refine the accuracy of the intelligent analysis engine. This process automatically optimizes the system, enabling more effective monitoring.

[0779] A concrete example of a prompt message could be, "Explain a mechanism that detects anomalies from a user's daily life data and notifies the user of their emotional state when stress levels are high." By entering this prompt message into the system, it becomes possible to discuss and deepen understanding of various aspects of the user's life.

[0780] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0781] Step 1:

[0782] Within the user's home, various sensors acquire power consumption, flow rate, audio, and video data in real time. This creates a detailed dataset of the user's daily life. The input is raw data transmitted from the sensors, and the output is data obtained in a format that is stored in temporary data storage.

[0783] Step 2:

[0784] The terminal receives data collected from the sensor and performs an initial integrity check. It identifies missing or inconsistent data and makes any necessary adjustments to ensure data reliability. Raw data from the sensor is used as input, and pre-processed data is obtained as output.

[0785] Step 3:

[0786] The server receives data sent from the terminal and applies denoising filters and interpolation algorithms. By reducing noisy data and imputing missing parts, it generates a clean and complete dataset. The input is the initial data sent from the terminal, and the output is the denoised and interpolated data.

[0787] Step 4:

[0788] The server analyzes pre-processed data using a generation AI model. It analyzes power usage patterns, voice tone, and facial expression data to learn lifestyle patterns and emotional states. The input is a clean dataset, and the output generates a model of the user's behavioral patterns and emotional states.

[0789] Step 5:

[0790] The server performs anomaly detection based on the generated model. It compares the analysis results with pre-set criteria and thresholds to identify behaviors and emotional states that deviate from normal patterns. Behavioral patterns and emotional state models are used as input, and the anomaly determination result is obtained as output.

[0791] Step 6:

[0792] The server sends an alert to pre-configured recipients if an anomaly is detected. The recipients' devices receive notifications, providing alerts regarding safety and health management. The input is the anomaly detection result, and the output is the notification sent to the recipient's device.

[0793] Step 7:

[0794] Users and recipients provide feedback to the system. Based on this feedback, the generated AI model is recalibrated, improving the accuracy of the analysis. Feedback information is used as input, and the improved model is obtained as output.

[0795] (Application Example 2)

[0796] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0797] In systems aimed at improving safety and security in daily life by comprehensively monitoring energy, resource, and water consumption within the home, as well as the emotional state of residents, and quickly detecting and notifying users of anomalies, conventional systems have the challenge of handling individual data separately, making unified monitoring difficult. Furthermore, there are concerns that delays in anomaly detection could lead to inadequate responses in situations requiring rapid action. In addition, there is a need to detect changes in emotional state early and enable appropriate feedback.

[0798] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0799] In this invention, the server includes means for acquiring energy, water, and fuel usage in a living space via one or more detection devices; means for preprocessing the acquired data by removing unwanted components and imputing missing values; and means for analyzing the preprocessed data using a machine learning model to learn the user's activity patterns. This makes it possible to detect abnormalities in the resident's emotional state and activity patterns in real time and promptly notify the resident's mobile communication terminal.

[0800] A "detection device" is a device used to measure the usage of electricity, water, and fuel within a living space. This includes sensors and meters.

[0801] "Removal of unwanted components" is a data preprocessing operation that removes noise and outliers contained within the data.

[0802] "Missing value imputation" is the operation of filling in missing values ​​in an incomplete dataset to generate complete data suitable for analysis.

[0803] A "machine learning model" is a set of algorithms used to learn patterns from data and perform analysis and predictions.

[0804] "Activity patterns" refer to a series of actions and tendencies related to a user's energy and resource consumption.

[0805] An "abnormality" is a sign that indicates behavior or a situation that deviates from normal activity patterns, and often requires a prompt response.

[0806] "Real-time notification" is a process that immediately transmits information about analyzed anomalies and emotional states to the user's mobile communication terminal.

[0807] The system implementing this invention centrally monitors the consumption of electricity, water, and fuel within the home, analyzes the emotional state of the residents to detect anomalies, and provides prompt notification. The server operates this system in the following specific manner.

[0808] First, the server collects energy and resource usage data from multiple detection devices. These include electricity meters, flow sensors, and gas detectors. These devices record specific consumption patterns within the living space in real time and transmit the data to the server.

[0809] Next, the server removes unwanted components and imputes missing values ​​from the received data to generate a clean dataset suitable for analysis. Specifically, it processes noise and anomalous missing data, transforming it into complete and consistent data.

[0810] Subsequently, the server operates a machine learning model based on the prepared data to learn the activity patterns of the residents. This model typically uses open-source libraries such as TensorFlow or PyTorch. The machine learning model analyzes past consumption data and acoustic information to detect stable behavioral patterns and emotional changes.

[0811] If an anomaly is detected, for example, if water usage increases significantly more than usual and acoustic data indicates feelings of anxiety, the server immediately activates a real-time notification function. This allows the user's mobile device to be notified of the anomaly. This notification is sent via a communication platform such as Firebase Cloud Messaging.

[0812] For example, if water leakage continues for an extended period, or if stress or anxiety is detected from the user's voice, a prompt message such as "A deviation from your normal lifestyle has been detected. Please check immediately." will be generated and sent as a notification.

[0813] In this way, the server functions as a system that enhances the safety and sense of security in the living space.

[0814] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0815] Step 1:

[0816] The server collects data from various detection devices installed in the home. Specifically, it acquires data in real time from devices such as power meters, flow sensors, and gas meters, and transmits it to the server. The input is raw data from these devices, and the output is unprocessed data aggregated on the server.

[0817] Step 2:

[0818] The server performs de-noise removal on the collected data. This process filters out data that may contain noise or outliers, generating a clean dataset. For example, if a sudden spike in power consumption is temporary noise, this is corrected. The input is the raw data from step 1, and the output is the de-noise-removed data.

[0819] Step 3:

[0820] The server performs missing value imputation on the data from Step 2. This step fills in data that was intermittently unavailable both temporally and correlatively. Artificial intelligence is used to imputate incomplete data based on historical data and trends. The input is clean data with missing values, and the output is a complete dataset.

[0821] Step 4:

[0822] The server uses a machine learning model to analyze the data from step 3. Here, it learns user activity patterns and emotional states to form standard patterns. Libraries such as TensorFlow and PyTorch are used for this analysis. The input is the complete dataset, and the output is the analyzed pattern information.

[0823] Step 5:

[0824] The server performs anomaly detection based on the analyzed pattern information. It sets specific thresholds and evaluates the data against them to identify anomalies. For example, this might apply if water usage significantly exceeds normal levels. The input consists of the analyzed pattern information and current data, while the output indicates whether an anomaly exists and provides detailed information about it.

[0825] Step 6:

[0826] If an anomaly is detected, the server immediately sends a real-time notification to the user's mobile device. This notification is sent using Firebase Cloud Messaging, etc. The prompt message sent will be, "A deviation from your normal daily routine has been detected. Please check immediately." The input is the anomaly detection information, and the output is the notification sent to the user's device.

[0827] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0828] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0829] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0830] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0831] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0832] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0833] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0834] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0835] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0836] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0837] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0838] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0839] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0840] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0841] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0842] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0843] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0844] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0845] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0846] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0847] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0848] The following is further disclosed regarding the embodiments described above.

[0849] (Claim 1)

[0850] A means for obtaining the usage status of electricity, water, and gas within a household via one or more sensors,

[0851] The acquired data includes means for noise reduction and missing value processing,

[0852] A means of learning user lifestyle patterns by analyzing pre-processed data using an artificial intelligence model,

[0853] A means for setting rules or thresholds to detect abnormal patterns based on the analyzed data,

[0854] A means of sending an alarm to a pre-configured recipient when an abnormal pattern is detected,

[0855] A means for adjusting the parameters of the artificial intelligence model based on user feedback,

[0856] A system that includes this.

[0857] (Claim 2)

[0858] The system according to claim 1, wherein the analyzed data includes acoustic data, and the system comprises means for classifying ambient noise or abnormal noise based on the acoustic data.

[0859] (Claim 3)

[0860] The system according to claim 1, wherein the alarm is a notification sent to the user's mobile device.

[0861] "Example 1"

[0862] (Claim 1)

[0863] Means for acquiring the usage status of energy resources in the environment via one or more sensing devices,

[0864] means for preprocessing the acquired information, including noise reduction and missing information completion,

[0865] A means of analyzing pre-processed information using a machine learning algorithm to learn the behavioral patterns of the monitored person,

[0866] Means for setting criteria or thresholds for detecting abnormal patterns based on the analyzed information,

[0867] A means for sending a warning signal to a pre-configured recipient when an abnormal pattern is detected,

[0868] A means for receiving a response from the monitored person and adjusting the parameters of the machine learning algorithm,

[0869] A system that includes this.

[0870] (Claim 2)

[0871] The system according to claim 1, wherein the analyzed information includes acoustic information, and the system comprises means for identifying ambient sounds or abnormal sounds based on the acoustic information.

[0872] (Claim 3)

[0873] The system according to claim 1, wherein the warning signal is a notification to the monitored person's mobile communication device.

[0874] "Application Example 1"

[0875] (Claim 1)

[0876] A means for obtaining household energy and liquid usage status via one or more detection means,

[0877] means for preprocessing the acquired information, including noise reduction and missing value processing,

[0878] A means of analyzing pre-processed information using a machine learning model to learn user behavior patterns,

[0879] Means for setting rules or criteria for detecting abnormal patterns based on the analyzed information,

[0880] A communication means that sends an alarm to a pre-configured recipient when an abnormal pattern is detected,

[0881] A means for adjusting the conditions of the machine learning model based on user feedback,

[0882] A means of detecting unauthorized external use to manage home security, recording and notifying of any anomalies,

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, wherein the analyzed information includes acoustic information, and the system comprises means for classifying everyday sounds or abnormal sounds based on the acoustic information.

[0886] (Claim 3)

[0887] The system according to claim 1, wherein the alarm is a notification sent to the user's mobile device.

[0888] "Example 2 of combining an emotion engine"

[0889] (Claim 1)

[0890] Means for acquiring indoor resource consumption and environmental data via one or more detection devices,

[0891] Means for applying removal filters and interpolation algorithms to preprocess the acquired data,

[0892] A means of learning individual behavioral patterns by analyzing pre-processed data using an intelligent analysis engine,

[0893] A means for setting criteria for detecting abnormal behaviors and emotional states based on analyzed data,

[0894] A means for transmitting warning information to a configured receiving device when an anomaly is detected,

[0895] A means for improving the analysis accuracy of the intelligent analysis engine based on feedback from individuals,

[0896] A system that includes this.

[0897] (Claim 2)

[0898] The system according to claim 1, wherein the analyzed data includes acoustic and visual data, and the system comprises means for identifying an emotional state based on this data.

[0899] (Claim 3)

[0900] The system according to claim 1, wherein the warning information is a notification sent to an individual's personal information terminal.

[0901] "Application example 2 when combining with an emotional engine"

[0902] (Claim 1)

[0903] Means for obtaining the usage of energy, water, and fuel in a living space via one or more detection devices,

[0904] The acquired data includes means for preprocessing by removing unnecessary components and imputing missing values,

[0905] A means of learning user activity patterns by analyzing preprocessed data using a machine learning model,

[0906] A means for setting criteria or thresholds for detecting abnormal operating patterns based on the analyzed data,

[0907] A means of sending a notification to a pre-configured recipient when abnormal operation is detected,

[0908] A means for adjusting the parameters of the machine learning model based on user feedback,

[0909] A means for transmitting abnormal emotional states and activity patterns in real time to the resident's mobile communication terminal,

[0910] A system that includes this.

[0911] (Claim 2)

[0912] The system according to claim 1, wherein the analyzed data includes acoustic information, and the system comprises means for classifying ambient sounds or abnormal sounds based on the acoustic information.

[0913] (Claim 3)

[0914] The system according to claim 1, wherein the notification is an alarm sent to the user's mobile device. [Explanation of symbols]

[0915] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for obtaining the usage status of electricity, water, and gas within a household via one or more sensors, The acquired data includes means for noise reduction and missing value processing, A means of learning user lifestyle patterns by analyzing pre-processed data using an artificial intelligence model, A means for setting rules or thresholds to detect abnormal patterns based on the analyzed data, A means of sending an alarm to a pre-configured recipient when an abnormal pattern is detected, A means for adjusting the parameters of the artificial intelligence model based on user feedback, A system that includes this.

2. The system according to claim 1, wherein the analyzed data includes acoustic data, and the system comprises means for classifying everyday sounds or abnormal sounds based on the acoustic data.

3. The system according to claim 1, wherein the alarm is a notification sent to the user's mobile device.

Images