system

Abstract

We provide the system. [Solution] A means of collecting data from sensors, A means of analyzing collected data and monitoring for anomalies, A means for generating and displaying a warning when an anomaly is detected, A means of providing a user-operable interface, A means of accumulating and analyzing logs of collected data, A means of sending notifications in an emergency, A system that includes means for encrypting and storing data.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, with the progress of an aging society, there is a social demand for improving the living safety and efficiency of the elderly and the elderly living alone. However, risk management associated with forgetting to turn off gas, electricity, and water or abnormalities in the home is not sufficiently carried out, and accidents may occur due to this. In addition, continuously monitoring these manually has become difficult due to the decline in the memory and attention of the elderly. Furthermore, there is a need for a system that can respond quickly to emergencies, but it is also necessary to consider privacy protection at the same time.

Means for Solving the Problems

[0005] To address the above challenges, the present invention provides a means for collecting, analyzing, and monitoring data from sensors. This enables automatic detection of abnormalities within the home, generating and displaying warnings. It also features an interface that allows users to remotely control the system, and by accumulating and analyzing logs of collected data, behavioral patterns can be understood. Furthermore, it includes a means for automatically sending notifications to pre-configured contacts in emergencies. These functions are implemented while protecting user privacy through means for securely storing encrypted data.

[0006] A "sensor" is a device that detects specific environmental conditions or physical states and outputs that information as an electrical signal.

[0007] A "data collection method" is a means that has the function of organizing information acquired from sensors and compiling it into a format that can be used for subsequent processing.

[0008] A "data analysis tool" is a tool that has the function of performing calculations and evaluations to detect specific patterns or anomalies based on collected information.

[0009] An "anomaly monitoring method" is a means that automatically detects abnormal situations by comparing the results of data analysis with pre-set criteria.

[0010] A "warning generation means" is a means that has the function of creating a message or signal to inform a person of danger when an anomaly is detected.

[0011] A "display means" is a device that conveys generated warnings or information to the user visually or audibly.

[0012] A "remote control interface" is a control window that allows users to operate equipment and systems from a physically distant location.

[0013] A "log storage method" is a means that has the function of saving collected and analyzed information as history and maintaining it in a state where it can be used for later analysis and verification.

[0014] An "analysis tool" is a means that uses accumulated data to analyze specific trends and patterns and extract necessary information.

[0015] An "emergency notification system" is a means of quickly notifying designated contacts of an alert or situation in the event of an unexpected incident or danger.

[0016] "Encryption" refers to technologies that protect privacy and confidentiality by converting data into a format that cannot be deciphered by third parties. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be described.

[0020] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The present invention is implemented as a smart home monitoring system including a sensor system and interface within the home. The configuration and functions of the system of the present invention will be described below with specific examples.

[0039] Sensor system configuration

[0040] First, sensors are installed throughout the home to detect gas, electricity, water, and door opening / closing. These sensors detect changes in status with high accuracy in real time and transmit the data to a server.

[0041] Server Role

[0042] The server periodically receives all data sent from the sensors using a specified sensor information protocol. This data is then stored, and an anomaly detection is performed by an analysis engine. The analysis engine compares the data with set criteria, for example, if the gas remains "on" for a certain period of time, to determine if an anomaly has occurred.

[0043] Warning generation and display

[0044] When an anomaly is detected, the server immediately generates a warning message. This warning information is sent to the home television terminal and displayed on the screen immediately. For example, if it is detected that the gas has been left on, a large message saying "The gas has been left on" will be displayed.

[0045] User operation means

[0046] Users can check for abnormal information and remotely control equipment via their TV remote or a dedicated smartphone app. For example, by launching the smartphone app and tapping the displayed warning, the gas can be turned off remotely.

[0047] Data accumulation and behavioral pattern analysis

[0048] Furthermore, the server continuously stores all operation logs and sensor data in a database. This data is used to analyze users' lifestyle patterns and usage trends. This is expected to improve not only safety but also energy efficiency.

[0049] Emergency notification function

[0050] In addition, in emergencies, the server automatically sends notifications to family members and caregivers via email or SMS. For example, if a gas leak is detected, relevant contacts will be immediately notified to prompt appropriate action.

[0051] Data security

[0052] Finally, all data is securely encrypted by the server and can only be accessed by those with specific access rights. In this way, the protection of personal information and the reliability of the system are ensured.

[0053] This integrated system provides strong support for elderly people to live independently and with peace of mind. Because this embodiment is flexible and expandable, it can be adapted to diverse home environments.

[0054] The following describes the processing flow.

[0055] Step 1:

[0056] The server periodically receives data from sensors within the home. This data includes information such as the on / off status of gas, electricity usage, water flow rate, and door open / closed status. The received data is temporarily stored in memory.

[0057] Step 2:

[0058] The server sends the received data to the analysis engine. The analysis engine compares this data to pre-configured criteria and evaluates whether there are any anomalies in real time. For example, if the gas remains "on" for a certain period of time or longer, it detects this as an anomaly.

[0059] Step 3:

[0060] When an anomaly is detected, the server immediately generates a warning message. This warning message includes details such as the nature of the anomaly, the time it occurred, and its location.

[0061] Step 4:

[0062] The server sends the generated warning message to the home television terminal. The data is quickly transferred via the transmission protocol and prepared for display on the television.

[0063] Step 5:

[0064] The device (television) displays the received warning message on its screen. The warning message is clearly displayed visually on the screen, and in some cases, an audio alert is also issued.

[0065] Step 6:

[0066] Users can review the displayed warnings and, if necessary, remotely control the relevant devices using their TV remote or smartphone app. For example, they can use a smartphone app to turn off the gas.

[0067] Step 7:

[0068] The server stores data from each operation and its associated sensors in a log database. The stored logs are used as foundational data for later analysis.

[0069] Step 8:

[0070] In the event of an emergency, the server will automatically send a notification to pre-registered contacts. This notification will include details of the anomaly and recommended actions to take.

[0071] Step 9:

[0072] Finally, the server encrypts all data and stores it in access-restricted storage to ensure user privacy and data security.

[0073] (Example 1)

[0074] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0075] Modern homes require efficient and safe living spaces using environmental control systems. However, if abnormalities are not detected early and promptly addressed, serious accidents and energy waste may occur. Furthermore, conventional systems have complex settings and are difficult for users to operate intuitively. This invention aims to solve these problems and provide a safer and more convenient home environment.

[0076] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0077] In this invention, the server includes means for collecting information from an environmental control device, means for analyzing the collected information and monitoring for anomalies, and means for providing an information terminal that can be remotely operated by the user. This enables the user to monitor anomalies in real time and respond quickly.

[0078] An "information terminal" is a device used by users to check data or perform remote operations.

[0079] An "environmental control device" is a device used to monitor the status of home appliances and residential equipment, and to control them as needed.

[0080] "Encoding" refers to the process of encrypting data to protect it from unauthorized access.

[0081] "History" refers to the records of collected data and operations, and is information used for analysis.

[0082] "Analysis means" refers to methods or devices for processing collected data and identifying specific patterns or anomalies.

[0083] "Communication" refers to a means of exchanging information with a distant location.

[0084] The system of this invention enables efficient and safe management in homes and offices by connecting to multiple environmental control devices. Its specific configuration and operation are described below.

[0085] Hardware and software configuration

[0086] The server is the primary component responsible for data processing and receives data from environmental control devices within the home. The server is equipped with a high-performance processor and large-capacity storage, and runs an analysis engine to analyze the received data. This analysis engine may include, for example, a real-time database management system or an AI-based anomaly detection algorithm.

[0087] A terminal is a device that allows a user to interface with a system. For example, wall-mounted touch panels or smartphone apps are used, enabling users to intuitively check information and perform operations.

[0088] The user is the entity that checks information about the system and, if necessary, performs remote operations. Users can use a TV remote control or a smartphone app to view warning messages and operate the device.

[0089] Data processing and data calculation

[0090] The server processes data received from the environmental control device using an analysis engine. During this process, the data is analyzed and evaluated based on rules for detecting anomalies. For example, if the gas remains "on" for a certain period of time, it is determined to be an anomaly, and an immediate notification is generated.

[0091] Data preservation and security are also considered, and the servers store all data encrypted. This ensures protection from unauthorized access.

[0092] Specific examples and prompt statements

[0093] For example, if a gas stove is unintentionally left on for an extended period, the server will detect this as an anomaly and immediately send a warning message to the user's device stating, "The gas has been left on." The user can then remotely turn off the gas by tapping the warning on the smartphone app.

[0094] Examples of input prompts for the generated AI model include, "Please explain how to be notified when a gas leak is detected in a smart home system," and "Please provide specific examples of data protocols from sensors to servers."

[0095] In this way, the system of the present invention can provide comprehensive support to offer a comfortable and safe living space.

[0096] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0097] Step 1:

[0098] The sensor monitors the status of environmental control devices within the home in real time and generates operational status data for each device. Specifically, it acquires information such as gas usage and door open / closed status. The input is status information from each environmental control device, and the output is detection data to be sent to the server.

[0099] Step 2:

[0100] The server receives data transmitted from sensors and processes it in its analysis engine. First, it receives status data from the environmental control device as input and records it in a database. The recorded data is then subjected to an anomaly detection algorithm by the analysis engine, which, for example, analyzes whether gas has been continuously used within a certain period of time to detect anomalies. The output is a warning message indicating whether or not an anomaly is present.

[0101] Step 3:

[0102] The server generates a warning message when an anomaly is detected and sends its contents to a display terminal in the home. The input is the result of the anomaly detection, and the output is a warning message such as "The gas has been left on." In actual operation, a communication protocol is used to send the message to the terminal.

[0103] Step 4:

[0104] The terminal immediately displays the received warning message on its screen. The input is the warning message sent from the server, and the output is the warning screen provided visually to the user. Specifically, the warning content is displayed prominently on the screen of a television or smartphone.

[0105] Step 5:

[0106] Users can view warning messages via a remote control or smartphone app and take necessary actions based on their content. The input is the content of the warning message, and the output is specific action instructions, such as turning off the gas. As for execution, users can remotely control the device by tapping operation buttons within the app.

[0107] Step 6:

[0108] The server stores all operation logs and sensor data in a database. The input consists of log data and sensor data for each operation, while the output is accumulated historical data. This historical data is later used to analyze user behavior patterns and usage trends.

[0109] Step 7:

[0110] The server analyzes lifestyle patterns from daily usage data based on its analysis engine, providing energy-saving suggestions and emergency response measures. Specifically, it analyzes historical data stored as input and generates reports showing energy-saving trends and predictive reports for abnormal trends as output. At this stage, it utilizes a generation AI model and performs advanced analysis using prompt messages.

[0111] This process allows the system to operate efficiently while maintaining the safety of the home environment.

[0112] (Application Example 1)

[0113] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0114] Modern homes require the effective management of information from multiple sensors to improve safety performance. However, conventional systems struggle to provide rapid notification and effective control in the event of anomalies, and are particularly inadequate in situations requiring real-time response, such as when away from home. We aim to solve this problem to achieve improved security and optimized energy efficiency.

[0115] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0116] In this invention, the server includes means for providing notifications of abnormalities to a mobile communication terminal or information display device, means for performing automated control based on specific conditions using an information analysis engine, and means for generating warning content using response generation technology to support rapid countermeasures. This enables monitoring of sensor information and rapid response even when away from home, improving home safety and efficiency.

[0117] A "sensor" is a device that detects physical or environmental changes and outputs that information as a digital or analog signal.

[0118] "Information" refers to data detected by sensors and the results of its analysis, which are used for monitoring and controlling the system.

[0119] An "analysis engine" is a software or hardware function that processes and analyzes collected data and detects anomalies by comparing it with specific conditions or criteria.

[0120] An "abnormality" refers to a state that deviates from the normal range set in the system, indicating a situation that requires immediate action or notification.

[0121] A "mobile communication terminal" is a portable device, such as a smartphone or tablet, that uses wireless communication to send and receive information.

[0122] A "notification" is a signal or message used to inform a user or administrator of detected anomalies or other important information.

[0123] "Control" refers to the actions taken to operate or adjust a system or device based on specific conditions in order to maintain or achieve a desired state.

[0124] "Response generation technology" is a technology that generates and provides appropriate warnings and guidance to users based on the received situation and data.

[0125] "Rapid response" refers to actions and processes that take appropriate measures without delay in response to abnormalities or emergencies.

[0126] A system for carrying out this invention includes a sensor device, a server, a user terminal, and an information display device. The sensor device detects various physical changes within the home and transmits the information to the server. The server analyzes the received information through a predetermined protocol and uses an analysis engine to detect anomalies based on specific criteria.

[0127] Based on the results of this analysis, the server provides appropriate notifications to mobile communication terminals and information display devices in the event of an anomaly. The notification content is generated using response generation technology and is immediately displayed on the receiving terminal. Based on these notifications, users can quickly take the necessary actions through a dedicated smartphone app or other mobile communication terminal. For example, if it is detected that the gas has been left on, the user can remotely operate the gas valve from their terminal to turn it off.

[0128] Furthermore, the server securely encrypts and stores all collected information logs, analyzing user behavior patterns and usage trends. This analysis is expected to lead to further security improvements and more efficient energy use. Specifically, it uses a generative AI model to learn the tendencies of anomalies and propose preventative measures.

[0129] As a concrete example, a user who is away from home might receive a notification that the gas in their home is still on. Upon receiving this notification, the user can use their mobile communication device to turn off the gas.

[0130] Another example of a generated AI prompt is: "I want to design a widget that clearly explains to the user about anomalies detected by the smart home security monitoring system. Please suggest what specific notification content this widget should display."

[0131] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0132] Step 1:

[0133] A sensor detects physical changes. The input is a change in the environment, and the output is a sensor signal. For example, a sensor might detect changes in gas or temperature, convert them into a digital signal, and send it to a server.

[0134] Step 2:

[0135] The server receives data transmitted from the sensor. The input is the sensor signal, and the output is data for analysis. The server converts the data into a predetermined format and prepares it for storage in the database.

[0136] Step 3:

[0137] The server uses a data analysis engine to detect anomalies. The input is data for analysis, and the output is a determination of whether or not an anomaly exists. As a condition setting, the server checks, for example, whether the gas usage time has exceeded a certain period of time.

[0138] Step 4:

[0139] When the server detects an anomaly, it generates a warning message using a generation AI model. The input is the anomaly detection result, and the output is the warning message. The server uses prompts and other elements to create a warning that is easy for the user to understand.

[0140] Step 5:

[0141] The server sends a warning message to the mobile communication terminal. The input is the warning message, and the output is a notification to the terminal. The server delivers the message to the user's smartphone via the network.

[0142] Step 6:

[0143] The user receives a notification from their device and takes the necessary action. The input is the warning message displayed on the device, and the output is the user's response. The user uses the app to perform actions such as turning off the gas.

[0144] Step 7:

[0145] The server accumulates and analyzes user behavior logs. The input is user behavior data, and the output is the result of behavioral pattern analysis. The server analyzes the data to identify future preventative measures and areas for improvement.

[0146] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0147] This invention combines a smart home system for managing safety within the home with an emotion engine that recognizes and responds to user emotions. This system is implemented with a complex configuration including sensors, a server, terminals, and the emotion engine.

[0148] Installation of the sensor system

[0149] First, various sensors are installed in the home. These sensors constantly monitor the status of gas, electricity, water, doors, etc., and transmit the fluctuating data to a server.

[0150] Server data processing

[0151] The server collects this sensor data in real time and generates an alert when an anomaly occurs. This alert is designed to notify the user immediately.

[0152] Embedding an emotion engine

[0153] A distinctive component of this invention is the incorporation of an emotion engine into the server, which analyzes the user's emotions. The emotion engine evaluates the user's emotional state using information obtained from sources such as voice tone, facial expressions, and a heart rate sensor, and reflects this in the operation of the entire system.

[0154] User operation and interface

[0155] Users interact with the system via in-home display devices or smartphone apps. Based on emotions detected by the emotion engine, the displayed interface can visually adapt. For example, if the user is stressed, the system will change to a simplified display or calming color scheme.

[0156] Emergency notifications and privacy protection

[0157] In the event of an emergency, the server adjusts the urgency of the notification based on the user's emotional state and sends an alert to pre-configured contacts. This ensures that the most relevant information is provided to the responder. The system encrypts and stores all data, including emotional data, to ensure user privacy.

[0158] This system design provides high safety and user experience, and functions as a solution to make the lives of the elderly and those living alone safer and more comfortable. This system not only detects anomalies but also takes into account the user's emotional state, enabling more nuanced and appropriate responses.

[0159] The following describes the processing flow.

[0160] Step 1:

[0161] The server receives data in real time from sensors installed in the home. This data includes gas usage, electricity consumption, water flow rate, and door open / closed status.

[0162] Step 2:

[0163] The server passes the received sensor data to the analysis engine for anomaly detection. By comparing it to set criteria, it detects anomalies such as when the gas remains "on" for a certain period of time.

[0164] Step 3:

[0165] When an anomaly is detected, the server activates the emotion engine and collects user emotion data. This data is collected from sources such as voice input devices, facial recognition cameras, and heart rate sensors.

[0166] Step 4:

[0167] The server adjusts the tone and urgency of warning messages based on emotional data analyzed by the emotion engine. For example, if the server determines that the user is relaxed, it will select a gentle warning sound.

[0168] Step 5:

[0169] The adjusted warning message is sent from the server to the home television terminal. The terminal displays the received message on the screen and adaptively adjusts the interface.

[0170] Step 6:

[0171] Users can use their TV remote or a smartphone app to view warning messages and, if necessary, remotely control the relevant device. For example, they can use the app to turn off the gas.

[0172] Step 7:

[0173] The server stores collected sensor and emotion data, as well as user operation history, in a database. This data is used to predict future behavior and improve the system.

[0174] Step 8:

[0175] In emergencies, the server sends a notification message to family members or caregivers that reflects emotional data. This message includes the user's current emotional state along with the most appropriate response information.

[0176] Step 9:

[0177] The server encrypts and stores all data, and access to the data is restricted to authorized users only. This ensures reliable data management while strictly protecting privacy.

[0178] (Example 2)

[0179] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0180] In modern homes, complex devices and systems are being introduced to improve safety and convenience, but these systems are limited to detecting and notifying of anomalies and lack the flexibility to respond based on the user's feelings and emotions. In particular, there is a growing demand for systems that can provide a sense of security in a more humane way, especially for the elderly and those living alone.

[0181] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0182] In this invention, the server includes means for acquiring information from sensors, means for analyzing the acquired information and monitoring for anomalies, and means for generating and displaying warnings when an anomaly is detected. This enables appropriate responses that take into account the user's emotional state and provides a safe and flexible home environment.

[0183] A "sensor" is a device that detects physical or environmental conditions and acquires corresponding information.

[0184] "Information" refers to data collected from sensors and other devices, which forms the basis for system analysis and control.

[0185] An "anomaly" refers to an event or phenomenon that deviates from the expected baseline or state, and is a condition that requires attention from the system.

[0186] A "warning" is a notification generated by the system when an anomaly is detected, intended to alert the user.

[0187] "User" refers to anyone who operates the system or uses its functions.

[0188] "Interactive format" refers to a form of user interface that allows users to exchange information with a system.

[0189] "Record keeping" refers to the storage of data that accumulates information acquired in the past and uses for later analysis and reference.

[0190] "Analysis" refers to the process of analyzing acquired information to identify and evaluate specific patterns or anomalies.

[0191] An "emotion analysis device" is part of a system that determines a user's emotional state based on information such as voice tone, facial expressions, and heart rate.

[0192] "Urgency" is an indicator of the importance of a notification being sent, and is used to determine the appropriate response speed and measures.

[0193] Encryption is a technology that transforms data to protect it from unauthorized access, making its contents unintelligible.

[0194] This smart home system is implemented using multiple hardware and software components. It primarily involves the coordinated operation of sensors, servers, and terminals.

[0195] Sensor configuration and data acquisition

[0196] The user monitors physical parameters using gas, electricity, water, and door sensors placed throughout the house. Each sensor has a built-in wireless communication module that transmits the observed data to the server in real time.

[0197] Server data analysis and anomaly detection

[0198] The server receives information transmitted from sensors and analyzes it by comparing it with pre-configured criteria and historical data. This analysis uses AI algorithms and database systems to effectively detect anomalies. If an anomaly is detected, the server generates a warning and immediately notifies the user.

[0199] Emotion analysis and system adaptation

[0200] The server incorporates an emotion analysis engine that analyzes the user's emotional state using voice tone, facial expressions, and heart rate data. The results of this analysis are reflected in system operation, and the interface is dynamically adapted by the terminal according to the user's emotional state.

[0201] User interaction

[0202] Users can interact with the system through various in-home display devices and smartphone apps. The system adjusts the user interface based on analysis results, resulting in a simplified display and a pleasing color scheme.

[0203] Emergency notifications and data security

[0204] In emergencies, the server considers the urgency and sends notifications to the user and pre-configured contacts. Furthermore, all emotional and sensor data is encrypted and securely stored to protect user privacy.

[0205] Examples of specific cases and prompts for generative AI models.

[0206] For example, a prompt message for when a user is stressed might be something like, "If the user is feeling tired, change the lighting to a warmer tone and reduce the level of notifications." This instruction could then be input into the AI ​​model.

[0207] This system design provides a solution that enables users to live a safe and comfortable life.

[0208] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0209] Step 1:

[0210] The server acquires data from various sensors. It receives data on gas, electricity, water, and door status as input, and stores this data in its internal database. At this stage, the data is organized in chronological order and undergoes initial cleansing in preparation for later analysis.

[0211] Step 2:

[0212] The server analyzes the acquired data and detects anomalies. Using organized sensor data as input, it identifies abnormal values ​​by comparing them to normal values ​​using an AI algorithm. It generates an anomaly detection report as output and prepares for warnings. Specifically, if a sudden increase in power consumption or a gas leak is detected, the location is recorded in the report.

[0213] Step 3:

[0214] The server uses an emotion analysis engine to analyze the user's emotional state. It accepts user data such as voice tone, facial expressions, and heart rate as input, and analyzes this data to evaluate the user's current emotional state. An emotion state report is generated as output, and this report is reflected in the overall system operation.

[0215] Step 4:

[0216] The terminal adapts the user interface based on emotional state reports from the server. It receives emotional state reports as input and changes the display screen according to the user's emotions. The user is then provided with a visually adapted interface as output. Specifically, if the user is feeling stressed, the design changes to a more calming color scheme.

[0217] Step 5:

[0218] The server sends notifications to pre-configured contacts when an anomaly is detected or an emergency occurs. It uses anomaly detection reports and urgency data, which takes emotional state into account, as input to create notifications of the appropriate level and format. As output, it sends alerts to emergency contacts. These alerts will prompt immediate action, depending on the situation.

[0219] Step 6:

[0220] The server encrypts and stores all data to protect user privacy. It receives all accumulated data as input and processes it using AES 256-bit encryption technology. As output, the encrypted data is stored in secure storage, preventing unauthorized external access.

[0221] (Application Example 2)

[0222] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0223] While home safety management systems can immediately notify users when an anomaly is detected, they do not take into account the user's emotional state, potentially increasing anxiety and stress. Furthermore, they have the drawback of making it difficult to select appropriate countermeasures when an anomaly occurs.

[0224] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0225] In this invention, the server includes means for collecting information from sensors, means for analyzing the collected information and monitoring for anomalies, means for evaluating the user's emotional state using an emotion analysis device, and means for dynamically adjusting the content and display of notifications according to the user's emotional state. This makes it possible to respond in a more appropriate and stress-reducing way, taking the user's emotional state into consideration.

[0226] A "sensor" is a device that detects physical and chemical changes in the environment and collects the data.

[0227] "Means of collecting information" refers to methods and devices for collecting data using sensors, etc.

[0228] "Means for analysis and monitoring anomalies" refer to methods and devices for analyzing collected data and detecting deviations from standard values.

[0229] "Means for generating and displaying notifications" refers to a method or device for creating warning messages or similar messages and informing the user when an anomaly is detected.

[0230] "Means of providing operating means" refers to methods or devices that provide an interface that allows users to operate the system remotely.

[0231] "Means for accumulating and analyzing logs" refers to methods and devices for recording collected data and later performing statistical analysis.

[0232] "Means of sending notifications" refers to communication methods used to inform users of abnormal or emergency situations in real time.

[0233] "Means of encrypting and storing information" refers to methods and devices that use cryptographic technology to protect information in order to maintain data confidentiality.

[0234] An "emotion analysis device" is a device that analyzes a user's emotional state from data such as voice, facial expressions, and heart rate.

[0235] "Means for evaluating emotional state" refers to methods or devices for estimating a user's emotions and mental state using emotion analysis equipment.

[0236] "Means of dynamic adjustment" refer to methods or devices for changing the system's response in accordance with the user's emotional state.

[0237] The system to realize this application includes sensors, emotion analyzers, a server, and the user's smartphone. The server collects information from various sensors and monitors the status of gas, electricity, water, doors, etc. The sensors have the function of detecting changes in the environment and transmitting that information to the server.

[0238] The server analyzes the collected data and, if it detects an anomaly, generates a warning and notifies the user. The notification is displayed on the user's smartphone. The server uses an emotion analysis device to collect data such as voice tone, facial expressions, and heart rate to evaluate the user's emotional state. Based on the evaluation results, the content and display of the notification for the anomaly that occurred are dynamically adjusted according to the user's emotional state.

[0239] The hardware and software used include voice processing APIs, facial expression analysis APIs, and security sensor APIs. For example, voice data collected from a smartphone is analyzed by the voice processing API, and facial expression data is analyzed by the facial expression analysis API. The analysis results are sent to a server where anomaly detection and adjustment of notification content are performed.

[0240] For example, a user may be away from home when a gas leak is detected by a sensor. In this case, the server checks the user's emotional state and determines the appropriate form of notification. If the user is experiencing stress, the notification will be in a calm tone and, if necessary, will suggest an emergency shutoff of the gas supply.

[0241] By using a generative AI model, it is possible to generate notification text and countermeasures. An example of a prompt text would be: "Please describe the behavior of an AI app that adjusts the content of security warnings based on the user's emotional state. Specifically, please describe how the app will notify the user when an anomaly is detected while the user is relaxed."

[0242] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0243] Step 1:

[0244] The server collects environmental information from various sensors. The collected data includes gas, electricity, water, and door status. Inputs are numerical data obtained from each sensor, while outputs are stored in the server's database.

[0245] Step 2:

[0246] The server analyzes the collected data. The data analysis uses a method that detects anomalies by comparing the data to specified thresholds. The input is raw sensor data, and the server determines the presence or absence of anomalies based on the analysis. The output is that a warning flag is set if an anomaly is detected.

[0247] Step 3:

[0248] The server uses voice processing APIs and facial expression analysis APIs to collect emotional data from the user's smartphone. The input is the user's voice data and photos, and the output is the analyzed emotional state of the user (e.g., relaxed, stressed).

[0249] Step 4:

[0250] The server generates a notification based on the detected anomaly and the user's emotional state. The input is the anomaly data and the user's emotional state, and the output is the text and display format of the notification message. A generative AI model is used in this step to generate appropriate text based on the prompt.

[0251] Step 5:

[0252] The user's device receives and displays notifications from the server. The input is the notification message sent from the server, and the output is the warnings and suggestions displayed on the device. Specifically, if the user is experiencing stress, the notification will be presented in a calming tone and text.

[0253] Step 6:

[0254] Users remotely control the system from their smartphones as needed. Inputs are user requests for actions, and outputs are specific instructions sent to the server. For example, this could include pressing an emergency stop button for the gas supply.

[0255] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0256] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0257] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0258] [Second Embodiment]

[0259] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0260] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0261] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0262] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0263] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0264] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0265] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0266] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0267] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0268] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0269] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0270] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0271] The present invention is implemented as a smart home monitoring system including a sensor system and interface within the home. The configuration and functions of the system of the present invention will be described below with specific examples.

[0272] Sensor system configuration

[0273] First, sensors are installed throughout the home to detect gas, electricity, water, and door opening / closing. These sensors detect changes in status with high accuracy in real time and transmit the data to a server.

[0274] Server Role

[0275] The server periodically receives all data sent from the sensors using a specified sensor information protocol. This data is then stored, and an anomaly detection is performed by an analysis engine. The analysis engine compares the data with set criteria, for example, if the gas remains "on" for a certain period of time, to determine if an anomaly has occurred.

[0276] Warning generation and display

[0277] When an anomaly is detected, the server immediately generates a warning message. This warning information is sent to the home television terminal and displayed on the screen immediately. For example, if it is detected that the gas has been left on, a large message saying "The gas has been left on" will be displayed.

[0278] User operation means

[0279] Users can check for abnormal information and remotely control equipment via their TV remote or a dedicated smartphone app. For example, by launching the smartphone app and tapping the displayed warning, the gas can be turned off remotely.

[0280] Data accumulation and behavioral pattern analysis

[0281] Furthermore, the server continuously stores all operation logs and sensor data in a database. This data is used to analyze users' lifestyle patterns and usage trends. This is expected to improve not only safety but also energy efficiency.

[0282] Emergency notification function

[0283] In addition, in emergencies, the server automatically sends notifications to family members and caregivers via email or SMS. For example, if a gas leak is detected, relevant contacts will be immediately notified to prompt appropriate action.

[0284] Data security

[0285] Finally, all data is securely encrypted by the server, and only those with specific access rights can log in and view it. In this way, the protection of personal information and the reliability of the system are ensured.

[0286] Through the coordination of the entire system, strong support is provided for the elderly to live a secure and independent life. Since this embodiment has flexible scalability, it can adapt to various home environments.

[0287] The following describes the processing flow.

[0288] Step 1:

[0289] The server periodically receives data from sensors in the home. This data includes the on / off state of gas, electricity consumption, water flow rate, door opening / closing state, etc. The received data is temporarily stored in memory.

[0290] Step 2:

[0291] The server sends the received data to the analysis engine. The analysis engine compares this data with pre-set criteria and evaluates in real time whether there are any abnormalities. For example, if the gas remains "on" for more than a certain period of time, that state is detected as an abnormality.

[0292] Step 3:

[0293] When an abnormality is detected, the server immediately generates a warning message. This warning message includes details such as the content of the abnormality, the occurrence time, and the location.

[0294] Step 4:

[0295] The server sends the generated warning message to the TV terminal in the home. The data is quickly transferred via the transmission protocol and the display on the TV is prepared.

[0296] Step 5:

[0297] The terminal (TV) displays the received warning message on the screen. A visual warning message is clearly shown on the screen, and in some cases, an audio alert is also issued.

[0298] Step 6:

[0299] The user checks the displayed warning and remotely operates the corresponding device using the TV remote control or smartphone app as needed. For example, an operation to turn off the gas can be performed with the smartphone app.

[0300] Step 7:

[0301] The server accumulates the data of each operation and the related sensor data in the log database. The accumulated logs are used as basic data for later analysis.

[0302] Step 8:

[0303] In case of an emergency, the server automatically sends a notification to the pre-registered contacts. This notification includes details of the abnormality and recommended countermeasures.

[0304] Step 9:

[0305] Finally, the server encrypts all the data and stores it in a storage with access restrictions. This is to ensure the privacy and security of the user's data.

[0306] (Example 1)

[0307] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0308] Modern homes require efficient and safe living spaces using environmental control systems. However, if abnormalities are not detected early and promptly addressed, serious accidents and energy waste may occur. Furthermore, conventional systems have complex settings and are difficult for users to operate intuitively. This invention aims to solve these problems and provide a safer and more convenient home environment.

[0309] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0310] In this invention, the server includes means for collecting information from an environmental control device, means for analyzing the collected information and monitoring for anomalies, and means for providing an information terminal that can be remotely operated by the user. This enables the user to monitor anomalies in real time and respond quickly.

[0311] An "information terminal" is a device used by users to check data or perform remote operations.

[0312] An "environmental control device" is a device used to monitor the status of home appliances and residential equipment, and to control them as needed.

[0313] "Encoding" refers to the process of encrypting data to protect it from unauthorized access.

[0314] "History" refers to the records of collected data and operations, and is information used for analysis.

[0315] "Analysis means" refers to methods or devices for processing collected data and identifying specific patterns or anomalies.

[0316] "Communication" refers to a means of exchanging information with a distant location.

[0317] The system of this invention enables efficient and safe management in homes and offices by connecting to multiple environmental control devices. Its specific configuration and operation are described below.

[0318] Hardware and software configuration

[0319] The server is the primary component responsible for data processing and receives data from environmental control devices within the home. The server is equipped with a high-performance processor and large-capacity storage, and runs an analysis engine to analyze the received data. This analysis engine may include, for example, a real-time database management system or an AI-based anomaly detection algorithm.

[0320] A terminal is a device that allows a user to interface with a system. For example, wall-mounted touch panels or smartphone apps are used, enabling users to intuitively check information and perform operations.

[0321] The user is the entity that checks information about the system and, if necessary, performs remote operations. Users can use a TV remote control or a smartphone app to view warning messages and operate the device.

[0322] Data processing and data calculation

[0323] The server processes data received from the environmental control device using an analysis engine. During this process, the data is analyzed and evaluated based on rules for detecting anomalies. For example, if the gas remains "on" for a certain period of time, it is determined to be an anomaly, and an immediate notification is generated.

[0324] Data preservation and security are also considered, and the servers store all data encrypted. This ensures protection from unauthorized access.

[0325] Specific examples and prompt statements

[0326] For example, if a gas stove is unintentionally left on for an extended period, the server will detect this as an anomaly and immediately send a warning message to the user's device stating, "The gas has been left on." The user can then remotely turn off the gas by tapping the warning on the smartphone app.

[0327] Examples of input prompts for the generated AI model include, "Please explain how to be notified when a gas leak is detected in a smart home system," and "Please provide specific examples of data protocols from sensors to servers."

[0328] In this way, the system of the present invention can provide comprehensive support to offer a comfortable and safe living space.

[0329] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0330] Step 1:

[0331] The sensor monitors the status of environmental control devices within the home in real time and generates operational status data for each device. Specifically, it acquires information such as gas usage and door open / closed status. The input is status information from each environmental control device, and the output is detection data to be sent to the server.

[0332] Step 2:

[0333] The server receives data transmitted from sensors and processes it in its analysis engine. First, it receives status data from the environmental control device as input and records it in a database. The recorded data is then subjected to an anomaly detection algorithm by the analysis engine, which, for example, analyzes whether gas has been continuously used within a certain period of time to detect anomalies. The output is a warning message indicating whether or not an anomaly is present.

[0334] Step 3:

[0335] The server generates a warning message when an anomaly is detected and sends its contents to a display terminal in the home. The input is the result of the anomaly detection, and the output is a warning message such as "The gas has been left on." In actual operation, a communication protocol is used to send the message to the terminal.

[0336] Step 4:

[0337] The terminal immediately displays the received warning message on its screen. The input is the warning message sent from the server, and the output is the warning screen provided visually to the user. Specifically, the warning content is displayed prominently on the screen of a television or smartphone.

[0338] Step 5:

[0339] Users can view warning messages via a remote control or smartphone app and take necessary actions based on their content. The input is the content of the warning message, and the output is specific action instructions, such as turning off the gas. As for execution, users can remotely control the device by tapping operation buttons within the app.

[0340] Step 6:

[0341] The server stores all operation logs and sensor data in a database. The input consists of log data and sensor data for each operation, while the output is accumulated historical data. This historical data is later used to analyze user behavior patterns and usage trends.

[0342] Step 7:

[0343] The server analyzes lifestyle patterns from daily usage data based on its analysis engine, providing energy-saving suggestions and emergency response measures. Specifically, it analyzes historical data stored as input and generates reports showing energy-saving trends and predictive reports for abnormal trends as output. At this stage, it utilizes a generation AI model and performs advanced analysis using prompt messages.

[0344] This process allows the system to operate efficiently while maintaining the safety of the home environment.

[0345] (Application Example 1)

[0346] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0347] Modern homes require the effective management of information from multiple sensors to improve safety performance. However, conventional systems struggle to provide rapid notification and effective control in the event of anomalies, and are particularly inadequate in situations requiring real-time response, such as when away from home. We aim to solve this problem to achieve improved security and optimized energy efficiency.

[0348] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0349] In this invention, the server includes means for providing notifications of abnormalities to a mobile communication terminal or information display device, means for performing automated control based on specific conditions using an information analysis engine, and means for generating warning content using response generation technology to support rapid countermeasures. This enables monitoring of sensor information and rapid response even when away from home, improving home safety and efficiency.

[0350] A "sensor" is a device that detects physical or environmental changes and outputs that information as a digital or analog signal.

[0351] "Information" refers to data detected by sensors and the results of its analysis, which are used for monitoring and controlling the system.

[0352] An "analysis engine" is a software or hardware function that processes and analyzes collected data and detects anomalies by comparing it with specific conditions or criteria.

[0353] An "abnormality" refers to a state that deviates from the normal range set in the system, indicating a situation that requires immediate action or notification.

[0354] A "mobile communication terminal" is a portable device, such as a smartphone or tablet, that uses wireless communication to send and receive information.

[0355] A "notification" is a signal or message used to inform a user or administrator of detected anomalies or other important information.

[0356] "Control" refers to the actions taken to operate or adjust a system or device based on specific conditions in order to maintain or achieve a desired state.

[0357] "Response generation technology" is a technology that generates and provides appropriate warnings and guidance to users based on the received situation and data.

[0358] "Rapid response" refers to actions and processes that take appropriate measures without delay in response to abnormalities or emergencies.

[0359] A system for carrying out this invention includes a sensor device, a server, a user terminal, and an information display device. The sensor device detects various physical changes within the home and transmits the information to the server. The server analyzes the received information through a predetermined protocol and uses an analysis engine to detect anomalies based on specific criteria.

[0360] Based on the results of this analysis, the server provides appropriate notifications to mobile communication terminals and information display devices in the event of an anomaly. The notification content is generated using response generation technology and is immediately displayed on the receiving terminal. Based on these notifications, users can quickly take the necessary actions through a dedicated smartphone app or other mobile communication terminal. For example, if it is detected that the gas has been left on, the user can remotely operate the gas valve from their terminal to turn it off.

[0361] Furthermore, the server securely encrypts and stores all collected information logs, analyzing user behavior patterns and usage trends. This analysis is expected to lead to further security improvements and more efficient energy use. Specifically, it uses a generative AI model to learn the tendencies of anomalies and propose preventative measures.

[0362] As a concrete example, a user who is away from home might receive a notification that the gas in their home is still on. Upon receiving this notification, the user can use their mobile communication device to turn off the gas.

[0363] Another example of a generated AI prompt is: "I want to design a widget that clearly explains to the user about anomalies detected by the smart home security monitoring system. Please suggest what specific notification content this widget should display."

[0364] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0365] Step 1:

[0366] A sensor detects physical changes. The input is a change in the environment, and the output is a sensor signal. For example, a sensor might detect changes in gas or temperature, convert them into a digital signal, and send it to a server.

[0367] Step 2:

[0368] The server receives data transmitted from the sensor. The input is the sensor signal, and the output is data for analysis. The server converts the data into a predetermined format and prepares it for storage in the database.

[0369] Step 3:

[0370] The server uses a data analysis engine to detect anomalies. The input is data for analysis, and the output is a determination of whether or not an anomaly exists. As a condition setting, the server checks, for example, whether the gas usage time has exceeded a certain period of time.

[0371] Step 4:

[0372] When the server detects an anomaly, it generates a warning message using a generation AI model. The input is the anomaly detection result, and the output is the warning message. The server uses prompts and other elements to create a warning that is easy for the user to understand.

[0373] Step 5:

[0374] The server sends a warning message to the mobile communication terminal. The input is the warning message, and the output is a notification to the terminal. The server delivers the message to the user's smartphone via the network.

[0375] Step 6:

[0376] The user receives a notification from their device and takes the necessary action. The input is the warning message displayed on the device, and the output is the user's response. The user uses the app to perform actions such as turning off the gas.

[0377] Step 7:

[0378] The server accumulates and analyzes user behavior logs. The input is user behavior data, and the output is the result of behavioral pattern analysis. The server analyzes the data to identify future preventative measures and areas for improvement.

[0379] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0380] This invention combines a smart home system for managing safety within the home with an emotion engine that recognizes and responds to user emotions. This system is implemented with a complex configuration including sensors, a server, terminals, and the emotion engine.

[0381] Installation of the sensor system

[0382] First, various sensors are installed in the home. These sensors constantly monitor the status of gas, electricity, water, doors, etc., and transmit the fluctuating data to a server.

[0383] Server data processing

[0384] The server collects this sensor data in real time and generates an alert when an anomaly occurs. This alert is designed to notify the user immediately.

[0385] Embedding an emotion engine

[0386] A distinctive component of this invention is the incorporation of an emotion engine into the server, which analyzes the user's emotions. The emotion engine evaluates the user's emotional state using information obtained from sources such as voice tone, facial expressions, and a heart rate sensor, and reflects this in the operation of the entire system.

[0387] User operation and interface

[0388] Users interact with the system via in-home display devices or smartphone apps. Based on emotions detected by the emotion engine, the displayed interface can visually adapt. For example, if the user is stressed, the system will change to a simplified display or calming color scheme.

[0389] Emergency notifications and privacy protection

[0390] In the event of an emergency, the server adjusts the urgency of the notification based on the user's emotional state and sends an alert to pre-configured contacts. This ensures that the most relevant information is provided to the responder. The system encrypts and stores all data, including emotional data, to ensure user privacy.

[0391] This system design provides high safety and user experience, and functions as a solution to make the lives of the elderly and those living alone safer and more comfortable. This system not only detects anomalies but also takes into account the user's emotional state, enabling more nuanced and appropriate responses.

[0392] The following describes the processing flow.

[0393] Step 1:

[0394] The server receives data in real time from sensors installed in the home. This data includes gas usage, electricity consumption, water flow rate, and door open / closed status.

[0395] Step 2:

[0396] The server passes the received sensor data to the analysis engine for anomaly detection. By comparing it to set criteria, it detects anomalies such as when the gas remains "on" for a certain period of time.

[0397] Step 3:

[0398] When an anomaly is detected, the server activates the emotion engine and collects user emotion data. This data is collected from sources such as voice input devices, facial recognition cameras, and heart rate sensors.

[0399] Step 4:

[0400] The server adjusts the tone and urgency of warning messages based on emotional data analyzed by the emotion engine. For example, if the server determines that the user is relaxed, it will select a gentle warning sound.

[0401] Step 5:

[0402] The adjusted warning message is sent from the server to the home television terminal. The terminal displays the received message on the screen and adaptively adjusts the interface.

[0403] Step 6:

[0404] Users can use their TV remote or a smartphone app to view warning messages and, if necessary, remotely control the relevant device. For example, they can use the app to turn off the gas.

[0405] Step 7:

[0406] The server stores collected sensor and emotion data, as well as user operation history, in a database. This data is used to predict future behavior and improve the system.

[0407] Step 8:

[0408] In emergencies, the server sends a notification message to family members or caregivers that reflects emotional data. This message includes the user's current emotional state along with the most appropriate response information.

[0409] Step 9:

[0410] The server encrypts and stores all data, and access to the data is restricted to authorized users only. This ensures reliable data management while strictly protecting privacy.

[0411] (Example 2)

[0412] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0413] In modern homes, complex devices and systems are being introduced to improve safety and convenience, but these systems are limited to detecting and notifying of anomalies and lack the flexibility to respond based on the user's feelings and emotions. In particular, there is a growing demand for systems that can provide a sense of security in a more humane way, especially for the elderly and those living alone.

[0414] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0415] In this invention, the server includes means for acquiring information from sensors, means for analyzing the acquired information and monitoring for anomalies, and means for generating and displaying warnings when an anomaly is detected. This enables appropriate responses that take into account the user's emotional state and provides a safe and flexible home environment.

[0416] A "sensor" is a device that detects physical or environmental conditions and acquires corresponding information.

[0417] "Information" refers to data collected from sensors and other devices, which forms the basis for system analysis and control.

[0418] An "anomaly" refers to an event or phenomenon that deviates from the expected baseline or state, and is a condition that requires attention from the system.

[0419] A "warning" is a notification generated by the system when an anomaly is detected, intended to alert the user.

[0420] "User" refers to anyone who operates the system or uses its functions.

[0421] "Interactive format" refers to a form of user interface that allows users to exchange information with a system.

[0422] "Record keeping" refers to the storage of data that accumulates information acquired in the past and uses for later analysis and reference.

[0423] "Analysis" refers to the process of analyzing acquired information to identify and evaluate specific patterns or anomalies.

[0424] An "emotion analysis device" is part of a system that determines a user's emotional state based on information such as voice tone, facial expressions, and heart rate.

[0425] "Urgency" is an indicator of the importance of a notification being sent, and is used to determine the appropriate response speed and measures.

[0426] Encryption is a technology that transforms data to protect it from unauthorized access, making its contents unintelligible.

[0427] This smart home system is implemented using multiple hardware and software components. It primarily involves the coordinated operation of sensors, servers, and terminals.

[0428] Sensor configuration and data acquisition

[0429] The user monitors physical parameters using gas, electricity, water, and door sensors placed throughout the house. Each sensor has a built-in wireless communication module that transmits the observed data to the server in real time.

[0430] Server data analysis and anomaly detection

[0431] The server receives information transmitted from sensors and analyzes it by comparing it with pre-configured criteria and historical data. This analysis uses AI algorithms and database systems to effectively detect anomalies. If an anomaly is detected, the server generates a warning and immediately notifies the user.

[0432] Emotion analysis and system adaptation

[0433] The server incorporates an emotion analysis engine that analyzes the user's emotional state using voice tone, facial expressions, and heart rate data. The results of this analysis are reflected in system operation, and the interface is dynamically adapted by the terminal according to the user's emotional state.

[0434] User interaction

[0435] Users can interact with the system through various in-home display devices and smartphone apps. The system adjusts the user interface based on analysis results, resulting in a simplified display and a pleasing color scheme.

[0436] Emergency notifications and data security

[0437] In emergencies, the server considers the urgency and sends notifications to the user and pre-configured contacts. Furthermore, all emotional and sensor data is encrypted and securely stored to protect user privacy.

[0438] Examples of specific cases and prompts for generative AI models.

[0439] For example, a prompt message for when a user is stressed might be something like, "If the user is feeling tired, change the lighting to a warmer tone and reduce the level of notifications." This instruction could then be input into the AI ​​model.

[0440] This system design provides a solution that enables users to live a safe and comfortable life.

[0441] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0442] Step 1:

[0443] The server acquires data from various sensors. It receives data on gas, electricity, water, and door status as input, and stores this data in its internal database. At this stage, the data is organized in chronological order and undergoes initial cleansing in preparation for later analysis.

[0444] Step 2:

[0445] The server analyzes the acquired data and detects anomalies. Using organized sensor data as input, it identifies abnormal values ​​by comparing them to normal values ​​using an AI algorithm. It generates an anomaly detection report as output and prepares for warnings. Specifically, if a sudden increase in power consumption or a gas leak is detected, the location is recorded in the report.

[0446] Step 3:

[0447] The server uses an emotion analysis engine to analyze the user's emotional state. It accepts user data such as voice tone, facial expressions, and heart rate as input, and analyzes this data to evaluate the user's current emotional state. An emotion state report is generated as output, and this report is reflected in the overall system operation.

[0448] Step 4:

[0449] The terminal adapts the user interface based on emotional state reports from the server. It receives emotional state reports as input and changes the display screen according to the user's emotions. The user is then provided with a visually adapted interface as output. Specifically, if the user is feeling stressed, the design changes to a more calming color scheme.

[0450] Step 5:

[0451] The server sends notifications to pre-configured contacts when an anomaly is detected or an emergency occurs. It uses anomaly detection reports and urgency data, which takes emotional state into account, as input to create notifications of the appropriate level and format. As output, it sends alerts to emergency contacts. These alerts will prompt immediate action, depending on the situation.

[0452] Step 6:

[0453] The server encrypts and stores all data to protect user privacy. It receives all accumulated data as input and processes it using AES 256-bit encryption technology. As output, the encrypted data is stored in secure storage, preventing unauthorized external access.

[0454] (Application Example 2)

[0455] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0456] While home safety management systems can immediately notify users when an anomaly is detected, they do not take into account the user's emotional state, potentially increasing anxiety and stress. Furthermore, they have the drawback of making it difficult to select appropriate countermeasures when an anomaly occurs.

[0457] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0458] In this invention, the server includes means for collecting information from sensors, means for analyzing the collected information and monitoring for anomalies, means for evaluating the user's emotional state using an emotion analysis device, and means for dynamically adjusting the content and display of notifications according to the user's emotional state. This makes it possible to respond in a more appropriate and stress-reducing way, taking the user's emotional state into consideration.

[0459] A "sensor" is a device that detects physical and chemical changes in the environment and collects the data.

[0460] "Means of collecting information" refers to methods and devices for collecting data using sensors, etc.

[0461] "Means for analysis and monitoring anomalies" refer to methods and devices for analyzing collected data and detecting deviations from standard values.

[0462] "Means for generating and displaying notifications" refers to a method or device for creating warning messages or similar messages and informing the user when an anomaly is detected.

[0463] "Means of providing operating means" refers to methods or devices that provide an interface that allows users to operate the system remotely.

[0464] "Means for accumulating and analyzing logs" refers to methods and devices for recording collected data and later performing statistical analysis.

[0465] "Means of sending notifications" refers to communication methods used to inform users of abnormal or emergency situations in real time.

[0466] "Means of encrypting and storing information" refers to methods and devices that use cryptographic technology to protect information in order to maintain data confidentiality.

[0467] An "emotion analysis device" is a device that analyzes a user's emotional state from data such as voice, facial expressions, and heart rate.

[0468] "Means for evaluating emotional state" refers to methods or devices for estimating a user's emotions and mental state using emotion analysis equipment.

[0469] "Means of dynamic adjustment" refer to methods or devices for changing the system's response in accordance with the user's emotional state.

[0470] The system to realize this application includes sensors, emotion analyzers, a server, and the user's smartphone. The server collects information from various sensors and monitors the status of gas, electricity, water, doors, etc. The sensors have the function of detecting changes in the environment and transmitting that information to the server.

[0471] The server analyzes the collected data and, if it detects an anomaly, generates a warning and notifies the user. The notification is displayed on the user's smartphone. The server uses an emotion analysis device to collect data such as voice tone, facial expressions, and heart rate to evaluate the user's emotional state. Based on the evaluation results, the content and display of the notification for the anomaly that occurred are dynamically adjusted according to the user's emotional state.

[0472] The hardware and software used include voice processing APIs, facial expression analysis APIs, and security sensor APIs. For example, voice data collected from a smartphone is analyzed by the voice processing API, and facial expression data is analyzed by the facial expression analysis API. The analysis results are sent to a server where anomaly detection and adjustment of notification content are performed.

[0473] For example, a user may be away from home when a gas leak is detected by a sensor. In this case, the server checks the user's emotional state and determines the appropriate form of notification. If the user is experiencing stress, the notification will be in a calm tone and, if necessary, will suggest an emergency shutoff of the gas supply.

[0474] By using a generative AI model, it is possible to generate notification text and countermeasures. An example of a prompt text would be: "Please describe the behavior of an AI app that adjusts the content of security warnings based on the user's emotional state. Specifically, please describe how the app will notify the user when an anomaly is detected while the user is relaxed."

[0475] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0476] Step 1:

[0477] The server collects environmental information from various sensors. The collected data includes gas, electricity, water, and door status. Inputs are numerical data obtained from each sensor, while outputs are stored in the server's database.

[0478] Step 2:

[0479] The server analyzes the collected data. The data analysis uses a method that detects anomalies by comparing the data to specified thresholds. The input is raw sensor data, and the server determines the presence or absence of anomalies based on the analysis. The output is that a warning flag is set if an anomaly is detected.

[0480] Step 3:

[0481] The server uses voice processing APIs and facial expression analysis APIs to collect emotional data from the user's smartphone. The input is the user's voice data and photos, and the output is the analyzed emotional state of the user (e.g., relaxed, stressed).

[0482] Step 4:

[0483] The server generates a notification based on the detected anomaly and the user's emotional state. The input is the anomaly data and the user's emotional state, and the output is the text and display format of the notification message. A generative AI model is used in this step to generate appropriate text based on the prompt.

[0484] Step 5:

[0485] The user's device receives and displays notifications from the server. The input is the notification message sent from the server, and the output is the warnings and suggestions displayed on the device. Specifically, if the user is experiencing stress, the notification will be presented in a calming tone and text.

[0486] Step 6:

[0487] Users remotely control the system from their smartphones as needed. Inputs are user requests for actions, and outputs are specific instructions sent to the server. For example, this could include pressing an emergency stop button for the gas supply.

[0488] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0489] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0490] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0491] [Third Embodiment]

[0492] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0493] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0494] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0495] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0496] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0497] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0498] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0499] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0500] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0501] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0502] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0503] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0504] The present invention is implemented as a smart home monitoring system including a sensor system and interface within the home. The configuration and functions of the system of the present invention will be described below with specific examples.

[0505] Sensor system configuration

[0506] First, sensors are installed throughout the home to detect gas, electricity, water, and door opening / closing. These sensors detect changes in status with high accuracy in real time and transmit the data to a server.

[0507] Server Role

[0508] The server periodically receives all data sent from the sensors using a specified sensor information protocol. This data is then stored, and an anomaly detection is performed by an analysis engine. The analysis engine compares the data with set criteria, for example, if the gas remains "on" for a certain period of time, to determine if an anomaly has occurred.

[0509] Warning generation and display

[0510] When an anomaly is detected, the server immediately generates a warning message. This warning information is sent to the home television terminal and displayed on the screen immediately. For example, if it is detected that the gas has been left on, a large message saying "The gas has been left on" will be displayed.

[0511] User operation means

[0512] Users can check for abnormal information and remotely control equipment via their TV remote or a dedicated smartphone app. For example, by launching the smartphone app and tapping the displayed warning, the gas can be turned off remotely.

[0513] Data accumulation and behavioral pattern analysis

[0514] Furthermore, the server continuously stores all operation logs and sensor data in a database. This data is used to analyze users' lifestyle patterns and usage trends. This is expected to improve not only safety but also energy efficiency.

[0515] Emergency notification function

[0516] In addition, in emergencies, the server automatically sends notifications to family members and caregivers via email or SMS. For example, if a gas leak is detected, relevant contacts will be immediately notified to prompt appropriate action.

[0517] Data security

[0518] Finally, all data is securely encrypted by the server and can only be accessed by those with specific access rights. In this way, the protection of personal information and the reliability of the system are ensured.

[0519] This integrated system provides strong support for elderly people to live independently and with peace of mind. Because this embodiment is flexible and expandable, it can be adapted to diverse home environments.

[0520] The following describes the processing flow.

[0521] Step 1:

[0522] The server periodically receives data from sensors within the home. This data includes information such as the on / off status of gas, electricity usage, water flow rate, and door open / closed status. The received data is temporarily stored in memory.

[0523] Step 2:

[0524] The server sends the received data to the analysis engine. The analysis engine compares this data to pre-configured criteria and evaluates whether there are any anomalies in real time. For example, if the gas remains "on" for a certain period of time or longer, it detects this as an anomaly.

[0525] Step 3:

[0526] When an anomaly is detected, the server immediately generates a warning message. This warning message includes details such as the nature of the anomaly, the time it occurred, and its location.

[0527] Step 4:

[0528] The server sends the generated warning message to the home television terminal. The data is quickly transferred via the transmission protocol and prepared for display on the television.

[0529] Step 5:

[0530] The device (television) displays the received warning message on its screen. The warning message is clearly displayed visually on the screen, and in some cases, an audio alert is also issued.

[0531] Step 6:

[0532] Users can review the displayed warnings and, if necessary, remotely control the relevant devices using their TV remote or smartphone app. For example, they can use a smartphone app to turn off the gas.

[0533] Step 7:

[0534] The server stores data from each operation and its associated sensors in a log database. The stored logs are used as foundational data for later analysis.

[0535] Step 8:

[0536] In the event of an emergency, the server will automatically send a notification to pre-registered contacts. This notification will include details of the anomaly and recommended actions to take.

[0537] Step 9:

[0538] Finally, the server encrypts all data and stores it in access-restricted storage to ensure user privacy and data security.

[0539] (Example 1)

[0540] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0541] Modern homes require efficient and safe living spaces using environmental control systems. However, if abnormalities are not detected early and promptly addressed, serious accidents and energy waste may occur. Furthermore, conventional systems have complex settings and are difficult for users to operate intuitively. This invention aims to solve these problems and provide a safer and more convenient home environment.

[0542] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0543] In this invention, the server includes means for collecting information from an environmental control device, means for analyzing the collected information and monitoring for anomalies, and means for providing an information terminal that can be remotely operated by the user. This enables the user to monitor anomalies in real time and respond quickly.

[0544] An "information terminal" is a device used by users to check data or perform remote operations.

[0545] An "environmental control device" is a device used to monitor the status of home appliances and residential equipment, and to control them as needed.

[0546] "Encoding" refers to the process of encrypting data to protect it from unauthorized access.

[0547] "History" refers to the records of collected data and operations, and is information used for analysis.

[0548] "Analysis means" refers to methods or devices for processing collected data and identifying specific patterns or anomalies.

[0549] "Communication" refers to a means of exchanging information with a distant location.

[0550] The system of this invention enables efficient and safe management in homes and offices by connecting to multiple environmental control devices. Its specific configuration and operation are described below.

[0551] Hardware and software configuration

[0552] The server is the primary component responsible for data processing and receives data from environmental control devices within the home. The server is equipped with a high-performance processor and large-capacity storage, and runs an analysis engine to analyze the received data. This analysis engine may include, for example, a real-time database management system or an AI-based anomaly detection algorithm.

[0553] A terminal is a device that allows a user to interface with a system. For example, wall-mounted touch panels or smartphone apps are used, enabling users to intuitively check information and perform operations.

[0554] The user is the entity that checks information about the system and, if necessary, performs remote operations. Users can use a TV remote control or a smartphone app to view warning messages and operate the device.

[0555] Data processing and data calculation

[0556] The server processes data received from the environmental control device using an analysis engine. During this process, the data is analyzed and evaluated based on rules for detecting anomalies. For example, if the gas remains "on" for a certain period of time, it is determined to be an anomaly, and an immediate notification is generated.

[0557] Data preservation and security are also considered, and the servers store all data encrypted. This ensures protection from unauthorized access.

[0558] Specific examples and prompt statements

[0559] For example, if a gas stove is unintentionally left on for an extended period, the server will detect this as an anomaly and immediately send a warning message to the user's device stating, "The gas has been left on." The user can then remotely turn off the gas by tapping the warning on the smartphone app.

[0560] Examples of input prompts for the generated AI model include, "Please explain how to be notified when a gas leak is detected in a smart home system," and "Please provide specific examples of data protocols from sensors to servers."

[0561] In this way, the system of the present invention can provide comprehensive support to offer a comfortable and safe living space.

[0562] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0563] Step 1:

[0564] The sensor monitors the status of environmental control devices within the home in real time and generates operational status data for each device. Specifically, it acquires information such as gas usage and door open / closed status. The input is status information from each environmental control device, and the output is detection data to be sent to the server.

[0565] Step 2:

[0566] The server receives data transmitted from sensors and processes it in its analysis engine. First, it receives status data from the environmental control device as input and records it in a database. The recorded data is then subjected to an anomaly detection algorithm by the analysis engine, which, for example, analyzes whether gas has been continuously used within a certain period of time to detect anomalies. The output is a warning message indicating whether or not an anomaly is present.

[0567] Step 3:

[0568] The server generates a warning message when an anomaly is detected and sends its contents to a display terminal in the home. The input is the result of the anomaly detection, and the output is a warning message such as "The gas has been left on." In actual operation, a communication protocol is used to send the message to the terminal.

[0569] Step 4:

[0570] The terminal immediately displays the received warning message on its screen. The input is the warning message sent from the server, and the output is the warning screen provided visually to the user. Specifically, the warning content is displayed prominently on the screen of a television or smartphone.

[0571] Step 5:

[0572] Users can view warning messages via a remote control or smartphone app and take necessary actions based on their content. The input is the content of the warning message, and the output is specific action instructions, such as turning off the gas. As for execution, users can remotely control the device by tapping operation buttons within the app.

[0573] Step 6:

[0574] The server stores all operation logs and sensor data in a database. The input consists of log data and sensor data for each operation, while the output is accumulated historical data. This historical data is later used to analyze user behavior patterns and usage trends.

[0575] Step 7:

[0576] The server analyzes lifestyle patterns from daily usage data based on its analysis engine, providing energy-saving suggestions and emergency response measures. Specifically, it analyzes historical data stored as input and generates reports showing energy-saving trends and predictive reports for abnormal trends as output. At this stage, it utilizes a generation AI model and performs advanced analysis using prompt messages.

[0577] This process allows the system to operate efficiently while maintaining the safety of the home environment.

[0578] (Application Example 1)

[0579] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0580] Modern homes require the effective management of information from multiple sensors to improve safety performance. However, conventional systems struggle to provide rapid notification and effective control in the event of anomalies, and are particularly inadequate in situations requiring real-time response, such as when away from home. We aim to solve this problem to achieve improved security and optimized energy efficiency.

[0581] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0582] In this invention, the server includes means for providing notifications of abnormalities to a mobile communication terminal or information display device, means for performing automated control based on specific conditions using an information analysis engine, and means for generating warning content using response generation technology to support rapid countermeasures. This enables monitoring of sensor information and rapid response even when away from home, improving home safety and efficiency.

[0583] A "sensor" is a device that detects physical or environmental changes and outputs that information as a digital or analog signal.

[0584] "Information" refers to data detected by sensors and the results of its analysis, which are used for monitoring and controlling the system.

[0585] An "analysis engine" is a software or hardware function that processes and analyzes collected data and detects anomalies by comparing it with specific conditions or criteria.

[0586] An "abnormality" refers to a state that deviates from the normal range set in the system, indicating a situation that requires immediate action or notification.

[0587] A "mobile communication terminal" is a portable device, such as a smartphone or tablet, that uses wireless communication to send and receive information.

[0588] A "notification" is a signal or message used to inform a user or administrator of detected anomalies or other important information.

[0589] "Control" refers to the actions taken to operate or adjust a system or device based on specific conditions in order to maintain or achieve a desired state.

[0590] "Response generation technology" is a technology that generates and provides appropriate warnings and guidance to users based on the received situation and data.

[0591] "Rapid response" refers to actions and processes that take appropriate measures without delay in response to abnormalities or emergencies.

[0592] A system for carrying out this invention includes a sensor device, a server, a user terminal, and an information display device. The sensor device detects various physical changes within the home and transmits the information to the server. The server analyzes the received information through a predetermined protocol and uses an analysis engine to detect anomalies based on specific criteria.

[0593] Based on the results of this analysis, the server provides appropriate notifications to mobile communication terminals and information display devices in the event of an anomaly. The notification content is generated using response generation technology and is immediately displayed on the receiving terminal. Based on these notifications, users can quickly take the necessary actions through a dedicated smartphone app or other mobile communication terminal. For example, if it is detected that the gas has been left on, the user can remotely operate the gas valve from their terminal to turn it off.

[0594] Furthermore, the server securely encrypts and stores all collected information logs, analyzing user behavior patterns and usage trends. This analysis is expected to lead to further security improvements and more efficient energy use. Specifically, it uses a generative AI model to learn the tendencies of anomalies and propose preventative measures.

[0595] As a concrete example, a user who is away from home might receive a notification that the gas in their home is still on. Upon receiving this notification, the user can use their mobile communication device to turn off the gas.

[0596] Another example of a generated AI prompt is: "I want to design a widget that clearly explains to the user about anomalies detected by the smart home security monitoring system. Please suggest what specific notification content this widget should display."

[0597] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0598] Step 1:

[0599] A sensor detects physical changes. The input is a change in the environment, and the output is a sensor signal. For example, a sensor might detect changes in gas or temperature, convert them into a digital signal, and send it to a server.

[0600] Step 2:

[0601] The server receives data transmitted from the sensor. The input is the sensor signal, and the output is data for analysis. The server converts the data into a predetermined format and prepares it for storage in the database.

[0602] Step 3:

[0603] The server uses a data analysis engine to detect anomalies. The input is data for analysis, and the output is a determination of whether or not an anomaly exists. As a condition setting, the server checks, for example, whether the gas usage time has exceeded a certain period of time.

[0604] Step 4:

[0605] When the server detects an anomaly, it generates a warning message using a generation AI model. The input is the anomaly detection result, and the output is the warning message. The server uses prompts and other elements to create a warning that is easy for the user to understand.

[0606] Step 5:

[0607] The server sends a warning message to the mobile communication terminal. The input is the warning message, and the output is a notification to the terminal. The server delivers the message to the user's smartphone via the network.

[0608] Step 6:

[0609] The user receives a notification from their device and takes the necessary action. The input is the warning message displayed on the device, and the output is the user's response. The user uses the app to perform actions such as turning off the gas.

[0610] Step 7:

[0611] The server accumulates and analyzes user behavior logs. The input is user behavior data, and the output is the result of behavioral pattern analysis. The server analyzes the data to identify future preventative measures and areas for improvement.

[0612] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0613] This invention combines a smart home system for managing safety within the home with an emotion engine that recognizes and responds to user emotions. This system is implemented with a complex configuration including sensors, a server, terminals, and the emotion engine.

[0614] Installation of the sensor system

[0615] First, various sensors are installed in the home. These sensors constantly monitor the status of gas, electricity, water, doors, etc., and transmit the fluctuating data to a server.

[0616] Server data processing

[0617] The server collects this sensor data in real time and generates an alert when an anomaly occurs. This alert is designed to notify the user immediately.

[0618] Embedding an emotion engine

[0619] A distinctive component of this invention is the incorporation of an emotion engine into the server, which analyzes the user's emotions. The emotion engine evaluates the user's emotional state using information obtained from sources such as voice tone, facial expressions, and a heart rate sensor, and reflects this in the operation of the entire system.

[0620] User operation and interface

[0621] Users interact with the system via in-home display devices or smartphone apps. Based on emotions detected by the emotion engine, the displayed interface can visually adapt. For example, if the user is stressed, the system will change to a simplified display or calming color scheme.

[0622] Emergency notifications and privacy protection

[0623] In the event of an emergency, the server adjusts the urgency of the notification based on the user's emotional state and sends an alert to pre-configured contacts. This ensures that the most relevant information is provided to the responder. The system encrypts and stores all data, including emotional data, to ensure user privacy.

[0624] This system design provides high safety and user experience, and functions as a solution to make the lives of the elderly and those living alone safer and more comfortable. This system not only detects anomalies but also takes into account the user's emotional state, enabling more nuanced and appropriate responses.

[0625] The following describes the processing flow.

[0626] Step 1:

[0627] The server receives data in real time from sensors installed in the home. This data includes gas usage, electricity consumption, water flow rate, and door open / closed status.

[0628] Step 2:

[0629] The server passes the received sensor data to the analysis engine for anomaly detection. By comparing it to set criteria, it detects anomalies such as when the gas remains "on" for a certain period of time.

[0630] Step 3:

[0631] When an anomaly is detected, the server activates the emotion engine and collects user emotion data. This data is collected from sources such as voice input devices, facial recognition cameras, and heart rate sensors.

[0632] Step 4:

[0633] The server adjusts the tone and urgency of warning messages based on emotional data analyzed by the emotion engine. For example, if the server determines that the user is relaxed, it will select a gentle warning sound.

[0634] Step 5:

[0635] The adjusted warning message is sent from the server to the home television terminal. The terminal displays the received message on the screen and adaptively adjusts the interface.

[0636] Step 6:

[0637] Users can use their TV remote or a smartphone app to view warning messages and, if necessary, remotely control the relevant device. For example, they can use the app to turn off the gas.

[0638] Step 7:

[0639] The server stores collected sensor and emotion data, as well as user operation history, in a database. This data is used to predict future behavior and improve the system.

[0640] Step 8:

[0641] In emergencies, the server sends a notification message to family members or caregivers that reflects emotional data. This message includes the user's current emotional state along with the most appropriate response information.

[0642] Step 9:

[0643] The server encrypts and stores all data, and access to the data is restricted to authorized users only. This ensures reliable data management while strictly protecting privacy.

[0644] (Example 2)

[0645] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0646] In modern homes, complex devices and systems are being introduced to improve safety and convenience, but these systems are limited to detecting and notifying of anomalies and lack the flexibility to respond based on the user's feelings and emotions. In particular, there is a growing demand for systems that can provide a sense of security in a more humane way, especially for the elderly and those living alone.

[0647] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0648] In this invention, the server includes means for acquiring information from sensors, means for analyzing the acquired information and monitoring for anomalies, and means for generating and displaying warnings when an anomaly is detected. This enables appropriate responses that take into account the user's emotional state and provides a safe and flexible home environment.

[0649] A "sensor" is a device that detects physical or environmental conditions and acquires corresponding information.

[0650] "Information" refers to data collected from sensors and other devices, which forms the basis for system analysis and control.

[0651] An "anomaly" refers to an event or phenomenon that deviates from the expected baseline or state, and is a condition that requires attention from the system.

[0652] A "warning" is a notification generated by the system when an anomaly is detected, intended to alert the user.

[0653] "User" refers to anyone who operates the system or uses its functions.

[0654] "Interactive format" refers to a form of user interface that allows users to exchange information with a system.

[0655] "Record keeping" refers to the storage of data that accumulates information acquired in the past and uses for later analysis and reference.

[0656] "Analysis" refers to the process of analyzing acquired information to identify and evaluate specific patterns or anomalies.

[0657] An "emotion analysis device" is part of a system that determines a user's emotional state based on information such as voice tone, facial expressions, and heart rate.

[0658] "Urgency" is an indicator of the importance of a notification being sent, and is used to determine the appropriate response speed and measures.

[0659] Encryption is a technology that transforms data to protect it from unauthorized access, making its contents unintelligible.

[0660] This smart home system is implemented using multiple hardware and software components. It primarily involves the coordinated operation of sensors, servers, and terminals.

[0661] Sensor configuration and data acquisition

[0662] The user monitors physical parameters using gas, electricity, water, and door sensors placed throughout the house. Each sensor has a built-in wireless communication module that transmits the observed data to the server in real time.

[0663] Server data analysis and anomaly detection

[0664] The server receives information transmitted from sensors and analyzes it by comparing it with pre-configured criteria and historical data. This analysis uses AI algorithms and database systems to effectively detect anomalies. If an anomaly is detected, the server generates a warning and immediately notifies the user.

[0665] Emotion analysis and system adaptation

[0666] The server incorporates an emotion analysis engine that analyzes the user's emotional state using voice tone, facial expressions, and heart rate data. The results of this analysis are reflected in system operation, and the interface is dynamically adapted by the terminal according to the user's emotional state.

[0667] User interaction

[0668] Users can interact with the system through various in-home display devices and smartphone apps. The system adjusts the user interface based on analysis results, resulting in a simplified display and a pleasing color scheme.

[0669] Emergency notifications and data security

[0670] In emergencies, the server considers the urgency and sends notifications to the user and pre-configured contacts. Furthermore, all emotional and sensor data is encrypted and securely stored to protect user privacy.

[0671] Examples of specific cases and prompts for generative AI models.

[0672] For example, a prompt message for when a user is stressed might be something like, "If the user is feeling tired, change the lighting to a warmer tone and reduce the level of notifications." This instruction could then be input into the AI ​​model.

[0673] This system design provides a solution that enables users to live a safe and comfortable life.

[0674] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0675] Step 1:

[0676] The server acquires data from various sensors. It receives data on gas, electricity, water, and door status as input, and stores this data in its internal database. At this stage, the data is organized in chronological order and undergoes initial cleansing in preparation for later analysis.

[0677] Step 2:

[0678] The server analyzes the acquired data and detects anomalies. Using organized sensor data as input, it identifies abnormal values ​​by comparing them to normal values ​​using an AI algorithm. It generates an anomaly detection report as output and prepares for warnings. Specifically, if a sudden increase in power consumption or a gas leak is detected, the location is recorded in the report.

[0679] Step 3:

[0680] The server uses an emotion analysis engine to analyze the user's emotional state. It accepts user data such as voice tone, facial expressions, and heart rate as input, and analyzes this data to evaluate the user's current emotional state. An emotion state report is generated as output, and this report is reflected in the overall system operation.

[0681] Step 4:

[0682] The terminal adapts the user interface based on emotional state reports from the server. It receives emotional state reports as input and changes the display screen according to the user's emotions. The user is then provided with a visually adapted interface as output. Specifically, if the user is feeling stressed, the design changes to a more calming color scheme.

[0683] Step 5:

[0684] The server sends notifications to pre-configured contacts when an anomaly is detected or an emergency occurs. It uses anomaly detection reports and urgency data, which takes emotional state into account, as input to create notifications of the appropriate level and format. As output, it sends alerts to emergency contacts. These alerts will prompt immediate action, depending on the situation.

[0685] Step 6:

[0686] The server encrypts and stores all data to protect user privacy. It receives all accumulated data as input and processes it using AES 256-bit encryption technology. As output, the encrypted data is stored in secure storage, preventing unauthorized external access.

[0687] (Application Example 2)

[0688] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0689] While home safety management systems can immediately notify users when an anomaly is detected, they do not take into account the user's emotional state, potentially increasing anxiety and stress. Furthermore, they have the drawback of making it difficult to select appropriate countermeasures when an anomaly occurs.

[0690] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0691] In this invention, the server includes means for collecting information from sensors, means for analyzing the collected information and monitoring for anomalies, means for evaluating the user's emotional state using an emotion analysis device, and means for dynamically adjusting the content and display of notifications according to the user's emotional state. This makes it possible to respond in a more appropriate and stress-reducing way, taking the user's emotional state into consideration.

[0692] A "sensor" is a device that detects physical and chemical changes in the environment and collects the data.

[0693] "Means of collecting information" refers to methods and devices for collecting data using sensors, etc.

[0694] "Means for analysis and monitoring anomalies" refer to methods and devices for analyzing collected data and detecting deviations from standard values.

[0695] "Means for generating and displaying notifications" refers to a method or device for creating warning messages or similar messages and informing the user when an anomaly is detected.

[0696] "Means of providing operating means" refers to methods or devices that provide an interface that allows users to operate the system remotely.

[0697] "Means for accumulating and analyzing logs" refers to methods and devices for recording collected data and later performing statistical analysis.

[0698] "Means of sending notifications" refers to communication methods used to inform users of abnormal or emergency situations in real time.

[0699] "Means of encrypting and storing information" refers to methods and devices that use cryptographic technology to protect information in order to maintain data confidentiality.

[0700] An "emotion analysis device" is a device that analyzes a user's emotional state from data such as voice, facial expressions, and heart rate.

[0701] "Means for evaluating emotional state" refers to methods or devices for estimating a user's emotions and mental state using emotion analysis equipment.

[0702] "Means of dynamic adjustment" refer to methods or devices for changing the system's response in accordance with the user's emotional state.

[0703] The system to realize this application includes sensors, emotion analyzers, a server, and the user's smartphone. The server collects information from various sensors and monitors the status of gas, electricity, water, doors, etc. The sensors have the function of detecting changes in the environment and transmitting that information to the server.

[0704] The server analyzes the collected data and, if it detects an anomaly, generates a warning and notifies the user. The notification is displayed on the user's smartphone. The server uses an emotion analysis device to collect data such as voice tone, facial expressions, and heart rate to evaluate the user's emotional state. Based on the evaluation results, the content and display of the notification for the anomaly that occurred are dynamically adjusted according to the user's emotional state.

[0705] The hardware and software used include voice processing APIs, facial expression analysis APIs, and security sensor APIs. For example, voice data collected from a smartphone is analyzed by the voice processing API, and facial expression data is analyzed by the facial expression analysis API. The analysis results are sent to a server where anomaly detection and adjustment of notification content are performed.

[0706] For example, a user may be away from home when a gas leak is detected by a sensor. In this case, the server checks the user's emotional state and determines the appropriate form of notification. If the user is experiencing stress, the notification will be in a calm tone and, if necessary, will suggest an emergency shutoff of the gas supply.

[0707] By using a generative AI model, it is possible to generate notification text and countermeasures. An example of a prompt text would be: "Please describe the behavior of an AI app that adjusts the content of security warnings based on the user's emotional state. Specifically, please describe how the app will notify the user when an anomaly is detected while the user is relaxed."

[0708] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0709] Step 1:

[0710] The server collects environmental information from various sensors. The collected data includes gas, electricity, water, and door status. Inputs are numerical data obtained from each sensor, while outputs are stored in the server's database.

[0711] Step 2:

[0712] The server analyzes the collected data. The data analysis uses a method that detects anomalies by comparing the data to specified thresholds. The input is raw sensor data, and the server determines the presence or absence of anomalies based on the analysis. The output is that a warning flag is set if an anomaly is detected.

[0713] Step 3:

[0714] The server uses voice processing APIs and facial expression analysis APIs to collect emotional data from the user's smartphone. The input is the user's voice data and photos, and the output is the analyzed emotional state of the user (e.g., relaxed, stressed).

[0715] Step 4:

[0716] The server generates a notification based on the detected anomaly and the user's emotional state. The input is the anomaly data and the user's emotional state, and the output is the text and display format of the notification message. A generative AI model is used in this step to generate appropriate text based on the prompt.

[0717] Step 5:

[0718] The user's device receives and displays notifications from the server. The input is the notification message sent from the server, and the output is the warnings and suggestions displayed on the device. Specifically, if the user is experiencing stress, the notification will be presented in a calming tone and text.

[0719] Step 6:

[0720] Users remotely control the system from their smartphones as needed. Inputs are user requests for actions, and outputs are specific instructions sent to the server. For example, this could include pressing an emergency stop button for the gas supply.

[0721] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0722] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0723] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0724] [Fourth Embodiment]

[0725] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0726] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0727] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0728] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0729] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0730] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0731] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0732] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0733] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0734] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0735] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0736] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0737] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0738] The present invention is implemented as a smart home monitoring system including a sensor system and interface within the home. The configuration and functions of the system of the present invention will be described below with specific examples.

[0739] Sensor system configuration

[0740] First, sensors are installed throughout the home to detect gas, electricity, water, and door opening / closing. These sensors detect changes in status with high accuracy in real time and transmit the data to a server.

[0741] Server Role

[0742] The server periodically receives all data sent from the sensors using a specified sensor information protocol. This data is then stored, and an anomaly detection is performed by an analysis engine. The analysis engine compares the data with set criteria, for example, if the gas remains "on" for a certain period of time, to determine if an anomaly has occurred.

[0743] Warning generation and display

[0744] When an anomaly is detected, the server immediately generates a warning message. This warning information is sent to the home television terminal and displayed on the screen immediately. For example, if it is detected that the gas has been left on, a large message saying "The gas has been left on" will be displayed.

[0745] User operation means

[0746] Users can check for abnormal information and remotely control equipment via their TV remote or a dedicated smartphone app. For example, by launching the smartphone app and tapping the displayed warning, the gas can be turned off remotely.

[0747] Data accumulation and behavioral pattern analysis

[0748] Furthermore, the server continuously stores all operation logs and sensor data in a database. This data is used to analyze users' lifestyle patterns and usage trends. This is expected to improve not only safety but also energy efficiency.

[0749] Emergency notification function

[0750] In addition, in emergencies, the server automatically sends notifications to family members and caregivers via email or SMS. For example, if a gas leak is detected, relevant contacts will be immediately notified to prompt appropriate action.

[0751] Data security

[0752] Finally, all data is securely encrypted by the server and can only be accessed by those with specific access rights. In this way, the protection of personal information and the reliability of the system are ensured.

[0753] This integrated system provides strong support for elderly people to live independently and with peace of mind. Because this embodiment is flexible and expandable, it can be adapted to diverse home environments.

[0754] The following describes the processing flow.

[0755] Step 1:

[0756] The server periodically receives data from sensors within the home. This data includes information such as the on / off status of gas, electricity usage, water flow rate, and door open / closed status. The received data is temporarily stored in memory.

[0757] Step 2:

[0758] The server sends the received data to the analysis engine. The analysis engine compares this data to pre-configured criteria and evaluates whether there are any anomalies in real time. For example, if the gas remains "on" for a certain period of time or longer, it detects this as an anomaly.

[0759] Step 3:

[0760] When an anomaly is detected, the server immediately generates a warning message. This warning message includes details such as the nature of the anomaly, the time it occurred, and its location.

[0761] Step 4:

[0762] The server sends the generated warning message to the home television terminal. The data is quickly transferred via the transmission protocol and prepared for display on the television.

[0763] Step 5:

[0764] The device (television) displays the received warning message on its screen. The warning message is clearly displayed visually on the screen, and in some cases, an audio alert is also issued.

[0765] Step 6:

[0766] Users can review the displayed warnings and, if necessary, remotely control the relevant devices using their TV remote or smartphone app. For example, they can use a smartphone app to turn off the gas.

[0767] Step 7:

[0768] The server stores data from each operation and its associated sensors in a log database. The stored logs are used as foundational data for later analysis.

[0769] Step 8:

[0770] In the event of an emergency, the server will automatically send a notification to pre-registered contacts. This notification will include details of the anomaly and recommended actions to take.

[0771] Step 9:

[0772] Finally, the server encrypts all data and stores it in access-restricted storage to ensure user privacy and data security.

[0773] (Example 1)

[0774] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0775] Modern homes require efficient and safe living spaces using environmental control systems. However, if abnormalities are not detected early and promptly addressed, serious accidents and energy waste may occur. Furthermore, conventional systems have complex settings and are difficult for users to operate intuitively. This invention aims to solve these problems and provide a safer and more convenient home environment.

[0776] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0777] In this invention, the server includes means for collecting information from an environmental control device, means for analyzing the collected information and monitoring for anomalies, and means for providing an information terminal that can be remotely operated by the user. This enables the user to monitor anomalies in real time and respond quickly.

[0778] An "information terminal" is a device used by users to check data or perform remote operations.

[0779] An "environmental control device" is a device used to monitor the status of home appliances and residential equipment, and to control them as needed.

[0780] "Encoding" refers to the process of encrypting data to protect it from unauthorized access.

[0781] "History" refers to the records of collected data and operations, and is information used for analysis.

[0782] "Analysis means" refers to methods or devices for processing collected data and identifying specific patterns or anomalies.

[0783] "Communication" refers to a means of exchanging information with a distant location.

[0784] The system of this invention enables efficient and safe management in homes and offices by connecting to multiple environmental control devices. Its specific configuration and operation are described below.

[0785] Hardware and software configuration

[0786] The server is the primary component responsible for data processing and receives data from environmental control devices within the home. The server is equipped with a high-performance processor and large-capacity storage, and runs an analysis engine to analyze the received data. This analysis engine may include, for example, a real-time database management system or an AI-based anomaly detection algorithm.

[0787] A terminal is a device that allows a user to interface with a system. For example, wall-mounted touch panels or smartphone apps are used, enabling users to intuitively check information and perform operations.

[0788] The user is the entity that checks information about the system and, if necessary, performs remote operations. Users can use a TV remote control or a smartphone app to view warning messages and operate the device.

[0789] Data processing and data calculation

[0790] The server processes data received from the environmental control device using an analysis engine. During this process, the data is analyzed and evaluated based on rules for detecting anomalies. For example, if the gas remains "on" for a certain period of time, it is determined to be an anomaly, and an immediate notification is generated.

[0791] Data preservation and security are also considered, and the servers store all data encrypted. This ensures protection from unauthorized access.

[0792] Specific examples and prompt statements

[0793] For example, if a gas stove is unintentionally left on for an extended period, the server will detect this as an anomaly and immediately send a warning message to the user's device stating, "The gas has been left on." The user can then remotely turn off the gas by tapping the warning on the smartphone app.

[0794] Examples of input prompts for the generated AI model include, "Please explain how to be notified when a gas leak is detected in a smart home system," and "Please provide specific examples of data protocols from sensors to servers."

[0795] In this way, the system of the present invention can provide comprehensive support to offer a comfortable and safe living space.

[0796] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0797] Step 1:

[0798] The sensor monitors the status of environmental control devices within the home in real time and generates operational status data for each device. Specifically, it acquires information such as gas usage and door open / closed status. The input is status information from each environmental control device, and the output is detection data to be sent to the server.

[0799] Step 2:

[0800] The server receives data transmitted from sensors and processes it in its analysis engine. First, it receives status data from the environmental control device as input and records it in a database. The recorded data is then subjected to an anomaly detection algorithm by the analysis engine, which, for example, analyzes whether gas has been continuously used within a certain period of time to detect anomalies. The output is a warning message indicating whether or not an anomaly is present.

[0801] Step 3:

[0802] The server generates a warning message when an anomaly is detected and sends its contents to a display terminal in the home. The input is the result of the anomaly detection, and the output is a warning message such as "The gas has been left on." In actual operation, a communication protocol is used to send the message to the terminal.

[0803] Step 4:

[0804] The terminal immediately displays the received warning message on its screen. The input is the warning message sent from the server, and the output is the warning screen provided visually to the user. Specifically, the warning content is displayed prominently on the screen of a television or smartphone.

[0805] Step 5:

[0806] Users can view warning messages via a remote control or smartphone app and take necessary actions based on their content. The input is the content of the warning message, and the output is specific action instructions, such as turning off the gas. As for execution, users can remotely control the device by tapping operation buttons within the app.

[0807] Step 6:

[0808] The server stores all operation logs and sensor data in a database. The input consists of log data and sensor data for each operation, while the output is accumulated historical data. This historical data is later used to analyze user behavior patterns and usage trends.

[0809] Step 7:

[0810] The server analyzes lifestyle patterns from daily usage data based on its analysis engine, providing energy-saving suggestions and emergency response measures. Specifically, it analyzes historical data stored as input and generates reports showing energy-saving trends and predictive reports for abnormal trends as output. At this stage, it utilizes a generation AI model and performs advanced analysis using prompt messages.

[0811] This process allows the system to operate efficiently while maintaining the safety of the home environment.

[0812] (Application Example 1)

[0813] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0814] Modern homes require the effective management of information from multiple sensors to improve safety performance. However, conventional systems struggle to provide rapid notification and effective control in the event of anomalies, and are particularly inadequate in situations requiring real-time response, such as when away from home. We aim to solve this problem to achieve improved security and optimized energy efficiency.

[0815] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0816] In this invention, the server includes means for providing notifications of abnormalities to a mobile communication terminal or information display device, means for performing automated control based on specific conditions using an information analysis engine, and means for generating warning content using response generation technology to support rapid countermeasures. This enables monitoring of sensor information and rapid response even when away from home, improving home safety and efficiency.

[0817] A "sensor" is a device that detects physical or environmental changes and outputs that information as a digital or analog signal.

[0818] "Information" refers to data detected by sensors and the results of its analysis, which are used for monitoring and controlling the system.

[0819] An "analysis engine" is a software or hardware function that processes and analyzes collected data and detects anomalies by comparing it with specific conditions or criteria.

[0820] An "abnormality" refers to a state that deviates from the normal range set in the system, indicating a situation that requires immediate action or notification.

[0821] A "mobile communication terminal" is a portable device, such as a smartphone or tablet, that uses wireless communication to send and receive information.

[0822] A "notification" is a signal or message used to inform a user or administrator of detected anomalies or other important information.

[0823] "Control" refers to the actions taken to operate or adjust a system or device based on specific conditions in order to maintain or achieve a desired state.

[0824] "Response generation technology" is a technology that generates and provides appropriate warnings and guidance to users based on the received situation and data.

[0825] "Rapid response" refers to actions and processes that take appropriate measures without delay in response to abnormalities or emergencies.

[0826] A system for carrying out this invention includes a sensor device, a server, a user terminal, and an information display device. The sensor device detects various physical changes within the home and transmits the information to the server. The server analyzes the received information through a predetermined protocol and uses an analysis engine to detect anomalies based on specific criteria.

[0827] Based on the results of this analysis, the server provides appropriate notifications to mobile communication terminals and information display devices in the event of an anomaly. The notification content is generated using response generation technology and is immediately displayed on the receiving terminal. Based on these notifications, users can quickly take the necessary actions through a dedicated smartphone app or other mobile communication terminal. For example, if it is detected that the gas has been left on, the user can remotely operate the gas valve from their terminal to turn it off.

[0828] Furthermore, the server securely encrypts and stores all collected information logs, analyzing user behavior patterns and usage trends. This analysis is expected to lead to further security improvements and more efficient energy use. Specifically, it uses a generative AI model to learn the tendencies of anomalies and propose preventative measures.

[0829] As a concrete example, a user who is away from home might receive a notification that the gas in their home is still on. Upon receiving this notification, the user can use their mobile communication device to turn off the gas.

[0830] Another example of a generated AI prompt is: "I want to design a widget that clearly explains to the user about anomalies detected by the smart home security monitoring system. Please suggest what specific notification content this widget should display."

[0831] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0832] Step 1:

[0833] A sensor detects physical changes. The input is a change in the environment, and the output is a sensor signal. For example, a sensor might detect changes in gas or temperature, convert them into a digital signal, and send it to a server.

[0834] Step 2:

[0835] The server receives data transmitted from the sensor. The input is the sensor signal, and the output is data for analysis. The server converts the data into a predetermined format and prepares it for storage in the database.

[0836] Step 3:

[0837] The server uses a data analysis engine to detect anomalies. The input is data for analysis, and the output is a determination of whether or not an anomaly exists. As a condition setting, the server checks, for example, whether the gas usage time has exceeded a certain period of time.

[0838] Step 4:

[0839] When the server detects an anomaly, it generates a warning message using a generation AI model. The input is the anomaly detection result, and the output is the warning message. The server uses prompts and other elements to create a warning that is easy for the user to understand.

[0840] Step 5:

[0841] The server sends a warning message to the mobile communication terminal. The input is the warning message, and the output is a notification to the terminal. The server delivers the message to the user's smartphone via the network.

[0842] Step 6:

[0843] The user receives a notification from their device and takes the necessary action. The input is the warning message displayed on the device, and the output is the user's response. The user uses the app to perform actions such as turning off the gas.

[0844] Step 7:

[0845] The server accumulates and analyzes user behavior logs. The input is user behavior data, and the output is the result of behavioral pattern analysis. The server analyzes the data to identify future preventative measures and areas for improvement.

[0846] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0847] This invention combines a smart home system for managing safety within the home with an emotion engine that recognizes and responds to user emotions. This system is implemented with a complex configuration including sensors, a server, terminals, and the emotion engine.

[0848] Installation of the sensor system

[0849] First, various sensors are installed in the home. These sensors constantly monitor the status of gas, electricity, water, doors, etc., and transmit the fluctuating data to a server.

[0850] Server data processing

[0851] The server collects this sensor data in real time and generates an alert when an anomaly occurs. This alert is designed to notify the user immediately.

[0852] Embedding an emotion engine

[0853] A distinctive component of this invention is the incorporation of an emotion engine into the server, which analyzes the user's emotions. The emotion engine evaluates the user's emotional state using information obtained from sources such as voice tone, facial expressions, and a heart rate sensor, and reflects this in the operation of the entire system.

[0854] User operation and interface

[0855] Users interact with the system via in-home display devices or smartphone apps. Based on emotions detected by the emotion engine, the displayed interface can visually adapt. For example, if the user is stressed, the system will change to a simplified display or calming color scheme.

[0856] Emergency notifications and privacy protection

[0857] In the event of an emergency, the server adjusts the urgency of the notification based on the user's emotional state and sends an alert to pre-configured contacts. This ensures that the most relevant information is provided to the responder. The system encrypts and stores all data, including emotional data, to ensure user privacy.

[0858] This system design provides high safety and user experience, and functions as a solution to make the lives of the elderly and those living alone safer and more comfortable. This system not only detects anomalies but also takes into account the user's emotional state, enabling more nuanced and appropriate responses.

[0859] The following describes the processing flow.

[0860] Step 1:

[0861] The server receives data in real time from sensors installed in the home. This data includes gas usage, electricity consumption, water flow rate, and door open / closed status.

[0862] Step 2:

[0863] The server passes the received sensor data to the analysis engine for anomaly detection. By comparing it to set criteria, it detects anomalies such as when the gas remains "on" for a certain period of time.

[0864] Step 3:

[0865] When an anomaly is detected, the server activates the emotion engine and collects user emotion data. This data is collected from sources such as voice input devices, facial recognition cameras, and heart rate sensors.

[0866] Step 4:

[0867] The server adjusts the tone and urgency of warning messages based on emotional data analyzed by the emotion engine. For example, if the server determines that the user is relaxed, it will select a gentle warning sound.

[0868] Step 5:

[0869] The adjusted warning message is sent from the server to the home television terminal. The terminal displays the received message on the screen and adaptively adjusts the interface.

[0870] Step 6:

[0871] Users can use their TV remote or a smartphone app to view warning messages and, if necessary, remotely control the relevant device. For example, they can use the app to turn off the gas.

[0872] Step 7:

[0873] The server stores collected sensor and emotion data, as well as user operation history, in a database. This data is used to predict future behavior and improve the system.

[0874] Step 8:

[0875] In emergencies, the server sends a notification message to family members or caregivers that reflects emotional data. This message includes the user's current emotional state along with the most appropriate response information.

[0876] Step 9:

[0877] The server encrypts and stores all data, and access to the data is restricted to authorized users only. This ensures reliable data management while strictly protecting privacy.

[0878] (Example 2)

[0879] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0880] In modern homes, complex devices and systems are being introduced to improve safety and convenience, but these systems are limited to detecting and notifying of anomalies and lack the flexibility to respond based on the user's feelings and emotions. In particular, there is a growing demand for systems that can provide a sense of security in a more humane way, especially for the elderly and those living alone.

[0881] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0882] In this invention, the server includes means for acquiring information from sensors, means for analyzing the acquired information and monitoring for anomalies, and means for generating and displaying warnings when an anomaly is detected. This enables appropriate responses that take into account the user's emotional state and provides a safe and flexible home environment.

[0883] A "sensor" is a device that detects physical or environmental conditions and acquires corresponding information.

[0884] "Information" refers to data collected from sensors and other devices, which forms the basis for system analysis and control.

[0885] An "anomaly" refers to an event or phenomenon that deviates from the expected baseline or state, and is a condition that requires attention from the system.

[0886] A "warning" is a notification generated by the system when an anomaly is detected, intended to alert the user.

[0887] "User" refers to anyone who operates the system or uses its functions.

[0888] "Interactive format" refers to a form of user interface that allows users to exchange information with a system.

[0889] "Record keeping" refers to the storage of data that accumulates information acquired in the past and uses for later analysis and reference.

[0890] "Analysis" refers to the process of analyzing acquired information to identify and evaluate specific patterns or anomalies.

[0891] An "emotion analysis device" is part of a system that determines a user's emotional state based on information such as voice tone, facial expressions, and heart rate.

[0892] "Urgency" is an indicator of the importance of a notification being sent, and is used to determine the appropriate response speed and measures.

[0893] Encryption is a technology that transforms data to protect it from unauthorized access, making its contents unintelligible.

[0894] This smart home system is implemented using multiple hardware and software components. It primarily involves the coordinated operation of sensors, servers, and terminals.

[0895] Sensor configuration and data acquisition

[0896] The user monitors physical parameters using gas, electricity, water, and door sensors placed throughout the house. Each sensor has a built-in wireless communication module that transmits the observed data to the server in real time.

[0897] Server data analysis and anomaly detection

[0898] The server receives information transmitted from sensors and analyzes it by comparing it with pre-configured criteria and historical data. This analysis uses AI algorithms and database systems to effectively detect anomalies. If an anomaly is detected, the server generates a warning and immediately notifies the user.

[0899] Emotion analysis and system adaptation

[0900] The server incorporates an emotion analysis engine that analyzes the user's emotional state using voice tone, facial expressions, and heart rate data. The results of this analysis are reflected in system operation, and the interface is dynamically adapted by the terminal according to the user's emotional state.

[0901] User interaction

[0902] Users can interact with the system through various in-home display devices and smartphone apps. The system adjusts the user interface based on analysis results, resulting in a simplified display and a pleasing color scheme.

[0903] Emergency notifications and data security

[0904] In emergencies, the server considers the urgency and sends notifications to the user and pre-configured contacts. Furthermore, all emotional and sensor data is encrypted and securely stored to protect user privacy.

[0905] Examples of specific cases and prompts for generative AI models.

[0906] For example, a prompt message for when a user is stressed might be something like, "If the user is feeling tired, change the lighting to a warmer tone and reduce the level of notifications." This instruction could then be input into the AI ​​model.

[0907] This system design provides a solution that enables users to live a safe and comfortable life.

[0908] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0909] Step 1:

[0910] The server acquires data from various sensors. It receives data on gas, electricity, water, and door status as input, and stores this data in its internal database. At this stage, the data is organized in chronological order and undergoes initial cleansing in preparation for later analysis.

[0911] Step 2:

[0912] The server analyzes the acquired data and detects anomalies. Using organized sensor data as input, it identifies abnormal values ​​by comparing them to normal values ​​using an AI algorithm. It generates an anomaly detection report as output and prepares for warnings. Specifically, if a sudden increase in power consumption or a gas leak is detected, the location is recorded in the report.

[0913] Step 3:

[0914] The server uses an emotion analysis engine to analyze the user's emotional state. It accepts user data such as voice tone, facial expressions, and heart rate as input, and analyzes this data to evaluate the user's current emotional state. An emotion state report is generated as output, and this report is reflected in the overall system operation.

[0915] Step 4:

[0916] The terminal adapts the user interface based on emotional state reports from the server. It receives emotional state reports as input and changes the display screen according to the user's emotions. The user is then provided with a visually adapted interface as output. Specifically, if the user is feeling stressed, the design changes to a more calming color scheme.

[0917] Step 5:

[0918] The server sends notifications to pre-configured contacts when an anomaly is detected or an emergency occurs. It uses anomaly detection reports and urgency data, which takes emotional state into account, as input to create notifications of the appropriate level and format. As output, it sends alerts to emergency contacts. These alerts will prompt immediate action, depending on the situation.

[0919] Step 6:

[0920] The server encrypts and stores all data to protect user privacy. It receives all accumulated data as input and processes it using AES 256-bit encryption technology. As output, the encrypted data is stored in secure storage, preventing unauthorized external access.

[0921] (Application Example 2)

[0922] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0923] While home safety management systems can immediately notify users when an anomaly is detected, they do not take into account the user's emotional state, potentially increasing anxiety and stress. Furthermore, they have the drawback of making it difficult to select appropriate countermeasures when an anomaly occurs.

[0924] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0925] In this invention, the server includes means for collecting information from sensors, means for analyzing the collected information and monitoring for anomalies, means for evaluating the user's emotional state using an emotion analysis device, and means for dynamically adjusting the content and display of notifications according to the user's emotional state. This makes it possible to respond in a more appropriate and stress-reducing way, taking the user's emotional state into consideration.

[0926] A "sensor" is a device that detects physical and chemical changes in the environment and collects the data.

[0927] "Means of collecting information" refers to methods and devices for collecting data using sensors, etc.

[0928] "Means for analysis and monitoring anomalies" refer to methods and devices for analyzing collected data and detecting deviations from standard values.

[0929] "Means for generating and displaying notifications" refers to a method or device for creating warning messages or similar messages and informing the user when an anomaly is detected.

[0930] "Means of providing operating means" refers to methods or devices that provide an interface that allows users to operate the system remotely.

[0931] "Means for accumulating and analyzing logs" refers to methods and devices for recording collected data and later performing statistical analysis.

[0932] "Means of sending notifications" refers to communication methods used to inform users of abnormal or emergency situations in real time.

[0933] "Means of encrypting and storing information" refers to methods and devices that use cryptographic technology to protect information in order to maintain data confidentiality.

[0934] An "emotion analysis device" is a device that analyzes a user's emotional state from data such as voice, facial expressions, and heart rate.

[0935] "Means for evaluating emotional state" refers to methods or devices for estimating a user's emotions and mental state using emotion analysis equipment.

[0936] "Means of dynamic adjustment" refer to methods or devices for changing the system's response in accordance with the user's emotional state.

[0937] The system to realize this application includes sensors, emotion analyzers, a server, and the user's smartphone. The server collects information from various sensors and monitors the status of gas, electricity, water, doors, etc. The sensors have the function of detecting changes in the environment and transmitting that information to the server.

[0938] The server analyzes the collected data and, if it detects an anomaly, generates a warning and notifies the user. The notification is displayed on the user's smartphone. The server uses an emotion analysis device to collect data such as voice tone, facial expressions, and heart rate to evaluate the user's emotional state. Based on the evaluation results, the content and display of the notification for the anomaly that occurred are dynamically adjusted according to the user's emotional state.

[0939] The hardware and software used include voice processing APIs, facial expression analysis APIs, and security sensor APIs. For example, voice data collected from a smartphone is analyzed by the voice processing API, and facial expression data is analyzed by the facial expression analysis API. The analysis results are sent to a server where anomaly detection and adjustment of notification content are performed.

[0940] For example, a user may be away from home when a gas leak is detected by a sensor. In this case, the server checks the user's emotional state and determines the appropriate form of notification. If the user is experiencing stress, the notification will be in a calm tone and, if necessary, will suggest an emergency shutoff of the gas supply.

[0941] By using a generative AI model, it is possible to generate notification text and countermeasures. An example of a prompt text would be: "Please describe the behavior of an AI app that adjusts the content of security warnings based on the user's emotional state. Specifically, please describe how the app will notify the user when an anomaly is detected while the user is relaxed."

[0942] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0943] Step 1:

[0944] The server collects environmental information from various sensors. The collected data includes gas, electricity, water, and door status. Inputs are numerical data obtained from each sensor, while outputs are stored in the server's database.

[0945] Step 2:

[0946] The server analyzes the collected data. The data analysis uses a method that detects anomalies by comparing the data to specified thresholds. The input is raw sensor data, and the server determines the presence or absence of anomalies based on the analysis. The output is that a warning flag is set if an anomaly is detected.

[0947] Step 3:

[0948] The server uses voice processing APIs and facial expression analysis APIs to collect emotional data from the user's smartphone. The input is the user's voice data and photos, and the output is the analyzed emotional state of the user (e.g., relaxed, stressed).

[0949] Step 4:

[0950] The server generates a notification based on the detected anomaly and the user's emotional state. The input is the anomaly data and the user's emotional state, and the output is the text and display format of the notification message. A generative AI model is used in this step to generate appropriate text based on the prompt.

[0951] Step 5:

[0952] The user's device receives and displays notifications from the server. The input is the notification message sent from the server, and the output is the warnings and suggestions displayed on the device. Specifically, if the user is experiencing stress, the notification will be presented in a calming tone and text.

[0953] Step 6:

[0954] Users remotely control the system from their smartphones as needed. Inputs are user requests for actions, and outputs are specific instructions sent to the server. For example, this could include pressing an emergency stop button for the gas supply.

[0955] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0956] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0957] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0958] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0959] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0960] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0961] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0962] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0963] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0964] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0965] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0966] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0967] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0968] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0969] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0970] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0971] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0972] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0973] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0974] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0975] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0976] The following is further disclosed regarding the embodiments described above.

[0977] (Claim 1)

[0978] A means of collecting data from sensors,

[0979] A means of analyzing collected data and monitoring for anomalies,

[0980] A means for generating and displaying a warning when an anomaly is detected,

[0981] A means of providing a user-operable interface,

[0982] A means of accumulating and analyzing logs of collected data,

[0983] A means of sending notifications in an emergency,

[0984] A system that includes means for encrypting and storing data.

[0985] (Claim 2)

[0986] The system according to claim 1, which provides information to a home display device when an anomaly is detected based on data collected by a sensor.

[0987] (Claim 3)

[0988] The system according to claim 1, comprising an analysis means for analyzing behavioral patterns from logs of collected data.

[0989] "Example 1"

[0990] (Claim 1)

[0991] Means for collecting information from environmental control devices,

[0992] A means of analyzing the collected information and monitoring for anomalies,

[0993] A means of generating and displaying a notification when an anomaly is detected,

[0994] A means of providing information terminals that users can remotely control,

[0995] A means of accumulating and analyzing the history of collected information,

[0996] Means of transmitting communications in an emergency,

[0997] A system that includes means for encoding and storing information.

[0998] (Claim 2)

[0999] The system according to claim 1, which provides instructions to a display device when an abnormality is detected based on information collected by an environmental control device.

[1000] (Claim 3)

[1001] The system according to claim 1, comprising an analysis means for analyzing activity patterns from the history of collected information.

[1002] "Application Example 1"

[1003] (Claim 1)

[1004] Means of acquiring information from sensors,

[1005] A means of analyzing acquired information and monitoring for anomalies,

[1006] A means for generating and displaying a warning when an anomaly is detected,

[1007] Means for providing an interface that can be remotely controlled by an operator,

[1008] A means of recording, accumulating, and analyzing collected information,

[1009] A means of sending notifications in an emergency,

[1010] A means of encrypting and storing information,

[1011] Means for providing notifications in case of abnormality to a mobile communication terminal or information display device,

[1012] A means of performing automated control based on specific conditions using an information analysis engine,

[1013] A means of generating warning content using response generation technology to support rapid countermeasures,

[1014] A system that includes this.

[1015] (Claim 2)

[1016] The system according to claim 1, which provides notification on a home display device or mobile communication terminal when an abnormality is detected based on information acquired by a sensor.

[1017] (Claim 3)

[1018] The system according to claim 1, comprising an analysis means for analyzing behavioral trends from recorded information acquired, and performing automated control based on the analysis results.

[1019] "Example 2 of combining an emotion engine"

[1020] (Claim 1)

[1021] Means for acquiring information from sensors,

[1022] A means of monitoring anomalies by analyzing the acquired information,

[1023] A means for generating and displaying a warning when an anomaly is detected,

[1024] A means of providing an interactive format that can be remotely controlled by the user,

[1025] A means of recording, accumulating, and analyzing acquired information,

[1026] A means of analyzing a user's emotional state using an emotion analysis device,

[1027] A means for dynamically adjusting the display format based on the analyzed emotional state,

[1028] A means of sending notifications and adjusting the urgency level in emergencies,

[1029] A system that includes means for encrypting and storing information.

[1030] (Claim 2)

[1031] The system according to claim 1, which provides information to a home display device based on the emotional state analyzed by an emotion analysis device.

[1032] (Claim 3)

[1033] The system according to claim 1, comprising an analysis means for analyzing behavioral characteristics from the recorded information acquired.

[1034] "Application example 2 when combining with an emotional engine"

[1035] (Claim 1)

[1036] Means for collecting information from sensors,

[1037] A means of analyzing the collected information and monitoring for anomalies,

[1038] A means of generating and displaying a notification when an anomaly is detected,

[1039] A means of providing a remote control mechanism that can be operated by the user,

[1040] A means of accumulating and analyzing logs of collected information,

[1041] A means of sending notifications in an emergency,

[1042] A means of encrypting and storing information,

[1043] A means of evaluating a user's emotional state using an emotion analysis device,

[1044] A means of dynamically adjusting the content and display of notifications according to the user's emotional state,

[1045] A system that includes this.

[1046] (Claim 2)

[1047] The system according to claim 1, which provides information to a home display device when an abnormality is detected based on information collected by a sensor.

[1048] (Claim 3)

[1049] The system according to claim 1, comprising an analysis means for analyzing behavioral patterns from logs of collected information. [Explanation of Symbols]

[1050] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting data from sensors, A means of analyzing collected data and monitoring for anomalies, A means for generating and displaying a warning when an anomaly is detected, A means of providing a user-operable interface, A means of accumulating and analyzing logs of collected data, A means of sending notifications in an emergency, A system that includes means for encrypting and storing data.

2. The system according to claim 1, which provides information to a home display device when an abnormality is detected based on data collected by a sensor.

3. The system according to claim 1, comprising an analysis means for analyzing behavioral patterns from logs of collected data.

Images