system

Abstract

Provide a system. A method for collecting data from a plurality of sensors; means for transmitting said data to a server; means for the server to analyze the data and detect anomalies; means for sending a notification when the abnormality is detected; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For elderly people and people with disabilities to live safely at home, a rapid and appropriate response in emergencies is necessary. However, care providers and family members who are not always nearby can find it difficult to detect and respond to accidents or health problems early. For this reason, there is a need for technology to provide an environment where elderly people and people with disabilities can live safely. [Means for solving the problem]

[0005] The present invention provides a system that collects data from multiple sensors and transmits the sensor data to a server for analysis. This system includes a means for transmitting data detected by the sensors to a server, which then analyzes the data and detects abnormalities. If an abnormality is detected, the system has a means for immediately sending a notification, thereby enabling a rapid response in an emergency. The sensors can also include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors, and the server includes a means for receiving the data after encrypting it and storing the analysis results. This realizes a secure and highly reliable anomaly detection system.

[0006] A "sensor" is a device that detects physical environmental information and converts it into data.

[0007] "Data" refers to information collected by sensors, including values ​​such as temperature, motion, smoke, and gas.

[0008] "Server" refers to a computing device that receives data, analyzes it, and performs any necessary processing.

[0009] "Transmission" refers to the act of moving data from one point to another, and in the present invention, it particularly refers to the movement of sensor data from a terminal to a server.

[0010] "Analysis" refers to the computational process of using received data to detect specific patterns or anomalies.

[0011] "Abnormal" refers to a phenomenon or data that deviates from a normal pattern, and in the present invention refers to a situation that requires an emergency response.

[0012] "Notification" refers to the act of transmitting warnings or information to a third party (such as a care provider or family member) when an abnormality is detected.

[0013] A "motion sensor" is a type of sensor that detects the movement of an object.

[0014] A "temperature sensor" is a device that detects the surrounding temperature and converts it into data.

[0015] A "smoke sensor" is a type of sensor that detects the presence of smoke.

[0016] A "gas sensor" is a device that detects the concentration of a specific gas and converts it into data.

[0017] A "door sensor" is a type of sensor that detects whether a door is open or closed.

[0018] A "window sensor" is a type of sensor that detects whether a window is open or closed.

[0019] "Encryption" refers to the act of converting data using a specific algorithm to protect it so that it cannot be easily deciphered by third parties.

[0020] "Storage" refers to the act of keeping collected data and analysis results in a state where they can be referenced later. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention is a system that provides a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities.

[0043] System configuration

[0044] The system mainly consists of the following components:

[0045] 1. Sensors

[0046] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0047] 2. Data transmission device

[0048] The device sends the collected sensor data to a server at regular intervals, and this process is performed by encrypting the data for security reasons.

[0049] 3. Server

[0050] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0051] 4. Notification System

[0052] When the server detects an abnormality, it notifies the care provider or family member via email, SMS, voice call, etc. The notification will include the type of abnormality and recommended measures.

[0053] System Operation

[0054] The device first collects real-time data from multiple sensors placed throughout the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and door and window opening and closing status are continuously monitored.

[0055] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0056] If an abnormality is detected, the server immediately notifies care providers and family members in real time, enabling them to respond quickly to emergencies. For example, if an elderly person collapses or a fire breaks out, a notification with detailed information and a prompt to take action is sent.

[0057] Specific examples

[0058] For example, the system may be implemented in a home as follows: A motion sensor detects unusual activity in the hallway. Because this activity is unusual, the sensor collects data and the device sends it to a server. The server analyzes the received data and detects abnormal patterns. Based on the analysis, an abnormality is confirmed and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification includes the specific nature of the abnormality, the location information, and specific instructions for how to respond.

[0059] In this way, a system will be realized that provides an environment where the elderly and people with disabilities can live with peace of mind. This system will enable early detection of abnormalities and prompt response, reducing risks in emergencies.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The device initializes multiple sensors and checks that they are operating normally. This includes using the self-diagnostic function of each sensor and reading the initial values ​​to check for abnormalities.

[0063] Step 2:

[0064] The device periodically collects data from each sensor, such as a motion sensor that detects the presence or absence of movement, a temperature sensor that measures the indoor temperature, a smoke sensor that detects smoke, a gas sensor that detects gas leaks, and sensors that monitor the opening and closing of doors and windows, and obtains data in real time.

[0065] Step 3:

[0066] The device packages the collected sensor data at regular intervals and converts it into a format that the server can understand, such as JSON. The data is encrypted during this process.

[0067] Step 4:

[0068] The device then sends the packaged and encrypted data to the server, where the communication is performed using a secure protocol (e.g., HTTPS) to ensure the data is secure.

[0069] Step 5:

[0070] The server receives the data sent from the device and decrypts it, converting it into a format suitable for analysis.

[0071] Step 6:

[0072] The server preprocesses the received data, checking for missing values ​​and outliers, and imputes data as needed.

[0073] Step 7:

[0074] The server then runs AI analysis on the pre-processed data, using AI models (such as Isolation Forest) to detect anomalies and determine whether they match a specific pattern.

[0075] Step 8:

[0076] If an anomaly is detected, the server determines the nature of the anomaly and generates the necessary notification, which includes the type of anomaly, its location, and recommended actions to take.

[0077] Step 9:

[0078] The server then sends notifications to care providers and family members via email, SMS, or voice calls.

[0079] Step 10:

[0080] The server stores the analysis results and sensor data in a database, which can then be used for later analysis and report generation.

[0081] Step 11:

[0082] The server periodically generates and provides users with reports that include anomaly detection history, behavioral pattern analysis results, and daily health trends.

[0083] In this way, the system of the present invention achieves a series of processing flows from collecting sensor data to detecting anomalies, notifying, storing data, and generating reports.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] Previously, systems designed to provide a safe and secure residential environment for the elderly and people with disabilities lacked the technology to detect abnormalities, making it difficult to detect them in real time. Furthermore, security issues existed regarding the safe transmission and storage of collected sensor data. This made it difficult to respond quickly and reliably to emergencies, and user safety was not adequately ensured.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes means for receiving data from the terminal at regular intervals and encrypting the data before transmission, means for decrypting the received data, and means for analyzing the decrypted data using AI technology to detect anomalies, thereby enabling real-time anomaly detection and safe transmission and analysis of collected sensor data.

[0089] A "sensor" is a device that monitors environmental conditions and collects data.

[0090] A "motion sensor" is a sensor that detects the movement of an object and collects that information.

[0091] A "temperature sensor" is a sensor that measures the ambient temperature and collects that data.

[0092] A "smoke sensor" is a sensor that detects the presence of smoke and collects information about it.

[0093] A "gas sensor" is a sensor that detects the presence of harmful gases and collects information about them.

[0094] A "door sensor" is a sensor that detects whether a door is open or closed and collects that data.

[0095] A "window sensor" is a sensor that detects whether a window is open or closed and collects that data.

[0096] A "terminal" is a device that collects data from multiple sensors and transmits it to a server.

[0097] "Encryption" is the process of transforming data using special algorithms in order to transmit it securely.

[0098] A "server" is a computer system that receives and analyzes data sent from a terminal.

[0099] "Decryption" is the process of restoring encrypted data to its original state.

[0100] "AI technology" is a data analysis method that utilizes artificial intelligence, and is used to perform pattern recognition and anomaly detection.

[0101] A "database" is a system for storing analytical results and other information.

[0102] "Notification" is a means of sending information to care providers and family members when an abnormality is detected.

[0103] System configuration

[0104] This invention is a system for providing a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities.

[0105] Key Components

[0106] 1. Sensor group (terminal)

[0107] The device includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. These sensors monitor the environment in real time and collect data. For example, when a motion sensor detects movement, the information is collected immediately.

[0108] 2. Data transmission device (terminal)

[0109] The device sends the collected sensor data to a server at regular intervals. This process encrypts the data using an encryption algorithm such as AES (Advanced Encryption Standard). Specifically, security is ensured by sending encrypted data.

[0110] 3. Server

[0111] The server decrypts the received data and analyzes it using AI technology, such as Python machine learning libraries (e.g., TENSORFLOW (registered trademark)). The server compares the decrypted data with past data to detect abnormal patterns and behavioral patterns.

[0112] 4. Notification System (Server)

[0113] If the server detects an abnormality, it will notify the care provider or family in real time via email, SMS, voice call, etc. The notification will include a detailed description of the type of abnormality and countermeasures.

[0114] Specific examples

[0115] For example, the system may be implemented in a home as follows: A motion sensor detects unusual activity in the hallway. Because this activity is unusual, the sensor collects data and the device sends it to a server. The server analyzes the received data and detects abnormal patterns. Based on the analysis, an abnormality is confirmed and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification includes the specific nature of the abnormality, the location information, and specific instructions for how to respond.

[0116] Example prompts for generative AI models

[0117] "Please describe a sensor system that provides an environment where elderly people can live safely. The system collects data from motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors, analyzes it on a server, and detects abnormalities. For example, please include how to handle abnormal activity in the hallway."

[0118] This invention will realize a home environment where the elderly and people with disabilities can live safely. The system enables early detection of abnormalities and rapid response, reducing the risk of emergencies.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] Step 1:

[0121] Sensor data collection

[0122] The device collects data from motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors.

[0123] As input, real-time data from each sensor is provided, and the device temporarily stores this data in its memory.

[0124] As a specific example of operation, when a motion sensor detects movement in a hallway, the detected data is stored in memory.

[0125] As an output, all collected sensor data is stored in memory.

[0126] Step 2:

[0127] Data encryption and transmission

[0128] The terminal encrypts the collected sensor data using the AES encryption algorithm and sends it to the server.

[0129] The input is the sensor data collected in step 1.

[0130] An encryption process is performed to generate encrypted data, which is then sent from the terminal to the server.

[0131] As a specific example of operation, data from a motion sensor in a hallway is encrypted and sent to a server.

[0132] As output, the encrypted sensor data is sent to the server.

[0133] Step 3:

[0134] Receiving and Decrypting Data

[0135] The server receives the data sent from the terminal and decrypts the data through a decryption process.

[0136] As input, encrypted sensor data is provided.

[0137] A decoding process is performed to reconstruct the original sensor data.

[0138] As a specific example of operation, the server receives encrypted motion sensor data and decrypts it to obtain the original data.

[0139] As an output, the decoded sensor data is obtained.

[0140] Step 4:

[0141] Data analysis

[0142] The server analyzes the decoded sensor data using AI technology.

[0143] As input, the decoded sensor data is provided.

[0144] Data analysis involves comparing data with past data and using machine learning libraries (e.g., TensorFlow) to detect behavioral and anomalous patterns.

[0145] As a specific example of operation, the server analyzes data from a motion sensor and detects movements that differ from normal behavior patterns.

[0146] The output is the analysis result, which includes a judgment of normality or abnormality.

[0147] Step 5:

[0148] Anomaly detection

[0149] The server detects anomalies based on the results of data analysis.

[0150] As input, the analysis results obtained in step 4 are provided.

[0151] If an anomaly is detected, an anomaly flag is set in the analysis results.

[0152] As a specific example of operation, the server detects abnormal movement in a hallway and adds an abnormality flag.

[0153] The output is an anomaly detection result, which is an analysis result with an anomaly flag added.

[0154] Step 6:

[0155] Send notifications

[0156] If an abnormality is detected, the server sends a notification to the care provider or family.

[0157] As input, the analysis results in which an anomaly was detected are provided.

[0158] A notification will be generated and sent via email, SMS, voice call, etc. The notification will include the specific anomaly and its location.

[0159] As a specific example of operation, the server detects an abnormality and sends a notification of the abnormality to the care provider and family via email and SMS.

[0160] As an output, a notification is sent: a notification message containing the anomaly description and location information is generated and sent to the specified contacts.

[0161] (Application example 1)

[0162] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0163] In conventional residential environments, monitoring systems have been inadequate to ensure the safety of elderly and disabled people. Furthermore, there has been no system in brick-and-mortar stores that can detect and notify abnormalities in real time while ensuring the safety of elderly and disabled customers. The purpose of this invention is to solve these problems and provide an environment where elderly and disabled people can live and visit stores with peace of mind.

[0164] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0165] In this invention, the server includes means for collecting data from a plurality of sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for transmitting a notification when the abnormality is detected, means for detecting environmental abnormalities with sensors installed in the store, and means for transmitting a notification to a smartphone when the environmental abnormality is detected. This not only enables elderly people and people with disabilities to live in peace of mind, but also makes it possible to ensure the safety of elderly and disabled customers in physical stores in real time.

[0166] A "sensor" is a device that acquires environmental information in real time and collects it as data.

[0167] The "data transmission means" is a means for transmitting data collected by the sensor to the server.

[0168] A "server" is a computing device that analyzes collected data and detects anomalies.

[0169] The "anomaly detection means" is a means for detecting unusual patterns or anomalies from the analyzed data.

[0170] The "notification sending means" is a means for sending a notification to relevant parties when an abnormality is detected.

[0171] A "smartphone" is a portable multi-function information terminal that has communication capabilities and can run applications.

[0172] "Inside the store" refers to the interior space of a store that sells goods.

[0173] "Environmental abnormalities" refers to situations that deviate from normal conditions, such as high temperatures, smoke, gas leaks, etc.

[0174] A "motion sensor" is a sensor for detecting movement.

[0175] This system provides a safe and secure environment for the elderly and people with disabilities, and has monitoring and notification functions using sensors, servers, and smartphones. Furthermore, this system can also be applied to similar safety systems in brick-and-mortar stores.

[0176] System configuration

[0177] Sensor group

[0178] The system includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors and window sensors, which are installed at different locations in brick-and-mortar stores and homes to monitor environmental information in real time.

[0179] Data transmission method

[0180] The data collected from the sensors is encrypted before being sent to the server, ensuring data security.

[0181] server

[0182] The server decodes the received sensor data and analyzes it using AI technology. If an anomaly is detected as a result of the analysis, a notification is triggered. The server also has the function of storing the anomaly detection results in a database.

[0183] Notification sending method

[0184] If an abnormality is detected, the server will immediately send email, SMS, or in-app notifications to relevant people (e.g., care providers and family members) via smartphone applications, ensuring the safety of seniors and people with disabilities in real time.

[0185] Specific examples

[0186] Consider a specific example of implementing the present invention in a brick-and-mortar store via a smartphone application called "Safety Navi." Sensors installed at various points in the store collect environmental information in real time, and if an abnormality is detected, the data is sent to a server.

[0187] If the server analyzes the data and detects an abnormality, the abnormality information is sent to the customer's smartphone via the app. The notification is displayed in the following format:

[0188] Prompt Sentence Examples

[0189] "A high temperature alarm has been detected near the refrigerated display case. We are instructing customers in the area to move away immediately."

[0190] This will enable elderly and disabled customers to safely visit and enjoy shopping in physical stores, and if a situation requires attention, immediate notification will enable a prompt response.

[0191] Hardware and Software Used

[0192] Sensors: Motion sensor, temperature sensor, smoke sensor, gas sensor, door sensor, window sensor

[0193] Server: Used for data analysis and anomaly detection

[0194] Smartphone: Used to receive notifications

[0195] Software: Data analysis and notification system implementation using Python, specifically using libraries such as requests, smtplib, and json.

[0196] This will make it possible to provide residential and retail environments where elderly people and people with disabilities can live safely.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] Sensors collect real-time environmental information (motion, temperature, smoke, gas, door and window data) within physical stores and homes.

[0200] Input: Physical variables of the environment (motion, temperature, smoke, gases, open and closed doors and windows).

[0201] Data processing: Sensors convert physical variables into electrical signals, which are then generated as digital data.

[0202] Output: Digital data (environmental information).

[0203] Step 2:

[0204] The device sends the collected sensor data to the server at regular intervals, and the data is encrypted to maintain security.

[0205] Input: Sensor data (digital data).

[0206] Data Computation: The sensor data is encrypted and sent to the server using a secure channel.

[0207] Output: Encrypted sensor data.

[0208] Step 3:

[0209] The server decrypts the received encrypted data and analyzes the environmental data.

[0210] Input: Encrypted sensor data.

[0211] Data computation: Decrypting the data and using AI techniques to analyze it, for example, using generative AI models to detect abnormal patterns and compare them with normal behavioral patterns.

[0212] Output: Analysis results (anomaly detection results).

[0213] Step 4:

[0214] The server generates a notification if it detects an anomaly based on the analysis results.

[0215] Input: Analysis results.

[0216] Data calculation: Based on the analysis results, a notification message is generated if an abnormality is detected.

[0217] Output: Informational message.

[0218] Step 5:

[0219] The server sends the generated notification messages to the smartphones of care providers, family members, store managers, and customers.

[0220] Input: Notification message.

[0221] Data calculation: Send notification messages as email, SMS, or in-app notifications.

[0222] Output: The notification sent.

[0223] Step 6:

[0224] The user receives a notification message via their smartphone and takes necessary measures.

[0225] Input: Notification message.

[0226] Specific actions: The user checks the contents of the message and takes action according to the specific instructions, such as avoiding a specific location or contacting an administrator.

[0227] Output: User behavior (safety action).

[0228] This will allow the entire system to be integrated and provide a safe environment for seniors, people with disabilities, and brick-and-mortar store visitors.

[0229] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0230] This invention combines an emotion engine that recognizes the user's emotions with a system that provides a home environment where the elderly and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities. It also recognizes the user's emotional state and takes necessary measures.

[0231] System configuration

[0232] The system mainly consists of the following components:

[0233] 1. Sensors

[0234] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0235] 2. Data transmission device

[0236] The device sends the collected sensor data to a server at regular intervals, and this process is performed by encrypting the data for security reasons.

[0237] 3. Server

[0238] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0239] 4. Emotion Engine

[0240] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine uses image processing and voice analysis technologies to determine emotions from the user's facial expressions and vocalizations. This allows the server to monitor the user's mental state and notify care providers or family members as necessary.

[0241] 5. Notification System

[0242] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[0243] System Operation

[0244] The device first collects real-time data from multiple sensors placed in the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and the opening and closing status of doors and windows are continuously monitored.

[0245] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0246] The emotion engine analyzes the user's emotions from collected video and audio data. For example, it can determine whether the user is sad or angry from their facial expression, or recognize stress levels from their tone of voice.

[0247] If an abnormality or emotional change is detected, the server will immediately notify the care provider or family member in real time, enabling a rapid response. For example, if an elderly person collapses, a fire breaks out, or the user is experiencing extreme stress, a notification with detailed information and a prompt to take action will be sent.

[0248] Specific examples

[0249] For example, the system may be implemented in a home as follows: A motion sensor detects abnormal movement in the hallway. Furthermore, a facial recognition camera analyzes the user's facial expressions and detects a sad expression. This series of data is collected by the device, encrypted, and sent to a server. The server analyzes the data and detects abnormal movement and emotional changes. As a result, an abnormality is confirmed, and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification contains the specific details of the abnormality, the user's emotional state, and specific instructions for how to respond.

[0250] In this way, a system will be realized that provides an environment where elderly people and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, this system will also monitor the user's emotional state and contribute to maintaining mental health.

[0251] The processing flow will be explained below.

[0252] Step 1:

[0253] The device initializes multiple sensors (motion sensor, temperature sensor, smoke sensor, gas sensor, door sensor, window sensor) and checks their operation. Each sensor has a self-diagnostic function to check whether it is operating normally.

[0254] Step 2:

[0255] The device periodically collects data from each sensor. For example, a motion sensor acquires information whenever it detects movement. A temperature sensor measures the room temperature at regular intervals and updates the data. Smoke and gas sensors collect data immediately when they detect an abnormality.

[0256] Step 3:

[0257] The device converts the collected sensor data into JSON-formatted packets and encrypts them, preventing unauthorized access to the data during transmission.

[0258] Step 4:

[0259] The device sends encrypted sensor data to a server using a secure protocol such as HTTPS to protect the authenticity and privacy of the data.

[0260] Step 5:

[0261] The server decrypts the received data and converts it into JSON format data, which is then pre-processed for analysis.

[0262] Step 6:

[0263] The server performs anomaly detection on the received data, using an AI model (e.g., Isolation Forest) to determine whether the collected data is within normal ranges or if an abnormal pattern has been detected.

[0264] Step 7:

[0265] The server records the results of the anomaly detection in a database, and this data is saved as history for future reference.

[0266] Step 8:

[0267] The server uses image processing technology to analyze the user's facial expressions and recognize the user's emotions (e.g., joy, anger, sadness, etc.) based on the video data sent from the camera.

[0268] Step 9:

[0269] The server uses voice analysis technology to recognize emotions from the user's voice. Based on the audio data collected from the microphone, the server analyzes the tone and pitch of the user's voice to determine their emotional state.

[0270] Step 10:

[0271] The server combines the user's emotional data analyzed by the emotion engine with sensor data for a comprehensive analysis, which allows it to determine the user's overall health and safety.

[0272] Step 11:

[0273] The server notifies care providers and family members if an abnormality or emotional change is detected. The notification includes details of the detected abnormality, the user's emotional state, and recommended actions. Notifications are sent via email, SMS, or emergency call.

[0274] Step 12:

[0275] The server stores the analysis results and notification details in a database, which will be used for future analysis and report generation.

[0276] In this way, the system of the present invention specifically implements a series of processing flows including sensor data collection, anomaly detection, emotion recognition, notification, and data storage, thereby providing comprehensive support for the safety and mental health of users.

[0277] Example 2

[0278] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0279] To provide a residential environment where elderly people and people with disabilities can live safely, it is important to detect indoor environmental changes and abnormalities early and respond quickly. However, conventional systems only collect data and detect abnormalities using sensors, and are unable to grasp the user's emotional state, making it difficult to maintain mental health and provide appropriate care. The present invention aims to solve these problems and provide a residential environment where elderly people and people with disabilities can live more safely.

[0280] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0281] In this invention, the server includes means for collecting data from a plurality of sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for the server to analyze the user's emotions, and means for transmitting a notification when the server detects the abnormality or change in emotion. This enables early detection of abnormalities and rapid response, and maintenance of mental health by understanding the user's emotional state.

[0282] "Multiple sensors" refers to different types of sensors, such as motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.

[0283] "Means for collecting data" refers to the function of using multiple sensors to monitor the indoor environmental conditions and obtain data in real time.

[0284] "Means for transmitting data to a server" refers to the function for encrypting collected sensor data and transmitting it to a server via the Internet.

[0285] "Means for the server to analyze data and detect abnormalities" refers to the function in which the server uses AI technology to analyze the sensor data it receives and compare it with normal patterns to detect abnormalities early.

[0286] "Means for the server to analyze the user's emotions" refers to the function of determining the user's emotional state based on the video and audio data collected by the server using image processing technology and audio analysis technology.

[0287] "Means for sending notifications" refers to the function of notifying care providers and family members in real time via email, SMS, voice call, etc. when an abnormality or change in the user's emotions is detected.

[0288] This invention is a system for providing a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and can detect abnormalities and recognize the user's emotional state, taking necessary measures.

[0289] System configuration

[0290] The system consists of the following components:

[0291] 1. Sensors

[0292] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0293] 2. Data transmission device

[0294] The device sends the collected sensor data to a server at regular intervals. This process encrypts the data for security reasons. An encryption algorithm such as AES (Advanced Encryption Standard) is used.

[0295] 3. Server

[0296] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. AI models such as TensorFlow and PyTorch are used. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0297] 4. Emotion Engine

[0298] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine uses image processing technology (OpenCV, Dlib, etc.) and voice analysis technology (Google (registered trademark) voice recognition API, etc.) to determine emotions from the user's facial expressions and vocalizations.

[0299] 5. Notification System

[0300] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[0301] System Operation

[0302] The device first collects real-time data from multiple sensors placed in the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and the opening and closing status of doors and windows are continuously monitored.

[0303] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0304] The emotion engine analyzes the user's emotions from collected video and audio data. For example, it can determine whether the user is sad or angry from their facial expression, or recognize stress levels from their tone of voice.

[0305] If an abnormality or emotional change is detected, the server will immediately notify the care provider or family member in real time, enabling a rapid response. For example, if an elderly person collapses, a fire breaks out, or the user is experiencing extreme stress, a notification with detailed information and a prompt to take action will be sent.

[0306] Specific examples

[0307] For example, the system may be implemented in a home as follows: A motion sensor detects abnormal movement in the hallway. Furthermore, a facial recognition camera analyzes the user's facial expressions and detects a sad expression. This series of data is collected by the device, encrypted, and sent to a server. The server analyzes the data and detects abnormal movement and emotional changes. As a result, an abnormality is confirmed, and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification contains the specific details of the abnormality, the user's emotional state, and specific instructions for how to respond.

[0308] In this way, a system will be realized that provides an environment where elderly people and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, this system will also monitor the user's emotional state and contribute to maintaining mental health.

[0309] Prompt Sentence Examples

[0310] Below are some example prompts for input to a generative AI model:

[0311] Follow this prompt to explain in natural language the process of a system that provides a safe and secure living environment for the elderly and people with disabilities. This system uses multiple sensors to collect data and analyzes it on a server. It also uses an emotion engine to recognize the user's emotions and take necessary measures. Please provide specific steps and examples. Use the subject of either the server, device, or user, and use plain sentence endings.

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1:

[0314] The device collects data in real time from multiple sensors (motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.). The input data includes the motion, temperature, smoke, gas, and whether a door or window is open or closed, detected by the sensors. For example, when a motion sensor detects motion, that information is collected immediately and becomes input data. Specific operations include the process by which the device acquires data from the sensors. The output is the collected raw sensor data.

[0315] Step 2:

[0316] The terminal encrypts the collected sensor data. Here, an encryption algorithm such as AES (Advanced Encryption Standard) is used. The input is the raw sensor data collected in step 1. The specific operation includes the process of protecting the data using an encryption algorithm. The output is the encrypted sensor data.

[0317] Step 3:

[0318] The terminal sends the encrypted sensor data to the server via the Internet. The input is the sensor data encrypted in step 2. The specific operation includes a process of sending the data to the server using a communication protocol. The output is the encrypted data sent to the server.

[0319] Step 4:

[0320] The server receives and decrypts the encrypted data sent over the Internet. The input is the encrypted data sent in step 3. The specific operations include the process of decrypting the data using a decryption algorithm. The output is the decrypted sensor data.

[0321] Step 5:

[0322] The server analyzes the decoded sensor data and detects abnormal patterns. This is done using AI technology. The input is the sensor data decoded in step 4. Specific operations include analyzing the data using AI models such as TensorFlow and PyTorch to detect abnormalities. The output is the presence or absence of anomalies and the analysis results.

[0323] Step 6:

[0324] The server analyzes the user's emotions using the decoded video and audio data. The emotion engine uses image processing technologies (such as OpenCV and Dlib) and audio analysis technologies (such as Google's speech recognition API). The input is video and audio data. Specific operations include identifying the user's emotions from facial expressions and tone of voice. The output is the identified emotional state of the user.

[0325] Step 7:

[0326] The server notifies care providers and families if it detects anomalies or emotional changes. The inputs are the anomaly information detected in step 5 and the emotional state identified in step 6. Specific operations include sending information in real time via the notification system using email, SMS, or voice calls. The output is a notification sent to the care provider or family.

[0327] (Application example 2)

[0328] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] In order to provide an environment where the elderly and people with disabilities can live safely, it is important not only to detect abnormalities in the environment but also to monitor the user's emotional state and provide prompt care as needed. However, current systems are specialized in detecting abnormalities, and therefore have the problem of not being able to respond appropriately based on the user's emotional state.

[0330] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from multiple sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for transmitting a notification when the abnormality is detected, means for the server to analyze image data and voice data of the user to recognize emotions, and means for transmitting a notification according to the user's emotional state. This makes it possible to monitor the user's emotional state in addition to detecting abnormalities in the environment, and to respond quickly and appropriately.

[0331] The "means for collecting data from multiple sensors" is a device that acquires environmental data using various types of sensors such as motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors.

[0332] The "means for transmitting the data to the server" refers to a communication device and process for securely transmitting the collected sensor data to the server via the Internet or a dedicated line.

[0333] The "means by which the server analyzes the data and detects anomalies" refers to a system that uses artificial intelligence (AI) or machine learning algorithms to analyze data received on the server and detect unusual patterns or changes in the environment.

[0334] The "means for sending a notification when the abnormality is detected" refers to a system that, when the server confirms an abnormality, notifies pre-defined contacts (e.g., care providers or family members) of the details of the abnormality via email, SMS, voice call, etc.

[0335] "Means for recognizing emotions by analyzing user image and audio data" refers to artificial intelligence (AI) technology that analyzes video and audio data acquired through cameras and microphones, and determines the user's emotional state from their facial expressions and tone of voice.

[0336] The "means for sending notifications according to the user's emotional state" is a system that notifies care providers and family members in real time as needed based on the recognized emotions.

[0337] This invention is a system that provides a residential environment where elderly people and people with disabilities can live safely. Specifically, it collects data from multiple sensors, sends it to a server for analysis, and can detect abnormalities and recognize the user's emotional state. This system consists of the following main components:

[0338] 1. Sensors

[0339] The sensor group includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. These sensors are installed on the terminal to monitor the indoor environment in real time and collect data.

[0340] 2. Data transmission device

[0341] The device sends the collected sensor data to the server at regular intervals. The data is encrypted for security reasons.

[0342] 3. Server

[0343] The server analyzes the received sensor data. This analysis utilizes AI technology to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically saved in a database and have the function of sending prompt notifications in the event of an emergency.

[0344] 4. Emotion Engine

[0345] The server is equipped with an emotion engine that recognizes the user's emotions. This emotion engine uses image processing and voice analysis technologies to determine emotions from the user's facial expressions and voice, monitors the user's mental state, and notifies care providers or family members as necessary.

[0346] 5. Notification System

[0347] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[0348] As a concrete example of how this might work, the system might operate as follows:

[0349] In a home, when a motion sensor detects abnormal activity in the hallway, the data is collected by the device, encrypted, and sent to a server. A camera then captures the user's facial expressions, which an emotion engine analyzes to detect sadness. This data is then analyzed by the server, which identifies abnormalities and emotional changes. The server then sends an email and SMS notification to the care provider and family members. The notification describes the abnormality, the user's emotional state, and includes specific instructions for how to respond.

[0350] In this way, a system will be realized that provides an environment where the elderly and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, the system will also monitor the user's emotional state and contribute to maintaining mental health.

[0351] Example prompts for generative AI models

[0352] Read the code below and write a Python program that collects data from the camera and sensors, analyzes the user's emotions, and sends a notification if an abnormality occurs. Use OpenCV, requests, cryptography, and smtplib for the program.

[0353] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0354] Step 1:

[0355] The device collects real-time environmental data from multiple sensors (motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.). This input data includes measurements from each sensor (e.g., temperature, motion detection, gas concentration, etc.). The device temporarily stores the collected data and prepares it for the next step.

[0356] Step 2:

[0357] The device sends the collected sensor data to the server at regular intervals. During this process, the device encrypts the data using a secure communication protocol (e.g., TLS). The data includes measurements from each sensor, and the server receives and analyzes it.

[0358] Step 3:

[0359] The server analyzes the received sensor data using AI technology and machine learning algorithms. This analysis process identifies patterns in the data and compares them with normal behavior and environmental conditions to detect anomalies. The analysis results include whether an anomaly occurred and the type of anomaly detected.

[0360] Step 4:

[0361] The server analyzes the user's image and audio data collected from the camera and microphone to recognize emotions. The emotion engine uses image processing and audio analysis technologies to determine the user's emotional state (e.g., sadness, anger, happiness, etc.) from their facial expressions and tone of voice. The input for this process is video and audio data, and the output is the recognized emotional state.

[0362] Step 5:

[0363] The server continuously monitors the user's emotional state and detects abnormalities based on the analysis results. If an abnormality or a specific emotional state is detected, the server sends a notification to pre-defined care providers or family members. The notification includes the details of the abnormality, the user's emotional state, and countermeasures. The notification can be sent via email, SMS, voice call, or other methods.

[0364] Step 6:

[0365] The server stores the analysis results and notification history in a database. The stored data includes measurements from each sensor, the recognized emotional state, and the content of the sent notification. This allows the system to track the user's long-term behavioral patterns and emotional state and use them for future analysis and countermeasures.

[0366] Step 7:

[0367] Users can change system settings and update notification destinations as needed, allowing the system to respond flexibly to individual user needs. User input to change settings is sent to the server and reflected in the overall system operation.

[0368] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0369] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0370] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0371] [Second embodiment]

[0372] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0373] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0374] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0375] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0376] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0377] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0378] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0379] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0380] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0381] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0382] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0383] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0384] This invention is a system that provides a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities.

[0385] System configuration

[0386] The system mainly consists of the following components:

[0387] 1. Sensors

[0388] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0389] 2. Data transmission device

[0390] The device sends the collected sensor data to a server at regular intervals, and this process is performed by encrypting the data for security reasons.

[0391] 3. Server

[0392] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0393] 4. Notification System

[0394] When the server detects an abnormality, it notifies the care provider or family member via email, SMS, voice call, etc. The notification will include the type of abnormality and recommended measures.

[0395] System Operation

[0396] The device first collects real-time data from multiple sensors placed throughout the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and door and window opening and closing status are continuously monitored.

[0397] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0398] If an abnormality is detected, the server immediately notifies care providers and family members in real time, enabling them to respond quickly to emergencies. For example, if an elderly person collapses or a fire breaks out, a notification with detailed information and a prompt to take action is sent.

[0399] Specific examples

[0400] For example, the system may be implemented in a home as follows: A motion sensor detects unusual activity in the hallway. Because this activity is unusual, the sensor collects data and the device sends it to a server. The server analyzes the received data and detects abnormal patterns. Based on the analysis, an abnormality is confirmed and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification includes the specific nature of the abnormality, the location information, and specific instructions for how to respond.

[0401] In this way, a system will be realized that provides an environment where the elderly and people with disabilities can live with peace of mind. This system will enable early detection of abnormalities and prompt response, reducing risks in emergencies.

[0402] The processing flow will be explained below.

[0403] Step 1:

[0404] The device initializes multiple sensors and checks that they are operating normally. This includes using the self-diagnostic function of each sensor and reading the initial values ​​to check for abnormalities.

[0405] Step 2:

[0406] The device periodically collects data from each sensor, such as a motion sensor that detects the presence or absence of movement, a temperature sensor that measures the indoor temperature, a smoke sensor that detects smoke, a gas sensor that detects gas leaks, and sensors that monitor the opening and closing of doors and windows, and obtains data in real time.

[0407] Step 3:

[0408] The device packages the collected sensor data at regular intervals and converts it into a format that the server can understand, such as JSON. The data is encrypted during this process.

[0409] Step 4:

[0410] The device then sends the packaged and encrypted data to the server, where the communication is performed using a secure protocol (e.g., HTTPS) to ensure the data is secure.

[0411] Step 5:

[0412] The server receives the data sent from the device and decrypts it, converting it into a format suitable for analysis.

[0413] Step 6:

[0414] The server preprocesses the received data, checking for missing values ​​and outliers, and imputes data as needed.

[0415] Step 7:

[0416] The server then runs AI analysis on the pre-processed data, using AI models (such as Isolation Forest) to detect anomalies and determine whether they match a specific pattern.

[0417] Step 8:

[0418] If an anomaly is detected, the server determines the nature of the anomaly and generates the necessary notification, which includes the type of anomaly, its location, and recommended actions to take.

[0419] Step 9:

[0420] The server then sends notifications to care providers and family members via email, SMS, or voice calls.

[0421] Step 10:

[0422] The server stores the analysis results and sensor data in a database, which can then be used for later analysis and report generation.

[0423] Step 11:

[0424] The server periodically generates and provides users with reports that include anomaly detection history, behavioral pattern analysis results, and daily health trends.

[0425] In this way, the system of the present invention achieves a series of processing flows from collecting sensor data to detecting anomalies, notifying, storing data, and generating reports.

[0426] Example 1

[0427] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0428] Previously, systems designed to provide a safe and secure residential environment for the elderly and people with disabilities lacked the technology to detect abnormalities, making it difficult to detect them in real time. Furthermore, security issues existed regarding the safe transmission and storage of collected sensor data. This made it difficult to respond quickly and reliably to emergencies, and user safety was not adequately ensured.

[0429] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0430] In this invention, the server includes means for receiving data from the terminal at regular intervals and encrypting the data before transmission, means for decrypting the received data, and means for analyzing the decrypted data using AI technology to detect anomalies, thereby enabling real-time anomaly detection and safe transmission and analysis of collected sensor data.

[0431] A "sensor" is a device that monitors environmental conditions and collects data.

[0432] A "motion sensor" is a sensor that detects the movement of an object and collects that information.

[0433] A "temperature sensor" is a sensor that measures the ambient temperature and collects that data.

[0434] A "smoke sensor" is a sensor that detects the presence of smoke and collects information about it.

[0435] A "gas sensor" is a sensor that detects the presence of harmful gases and collects information about them.

[0436] A "door sensor" is a sensor that detects whether a door is open or closed and collects that data.

[0437] A "window sensor" is a sensor that detects whether a window is open or closed and collects that data.

[0438] A "terminal" is a device that collects data from multiple sensors and transmits it to a server.

[0439] "Encryption" is the process of transforming data using special algorithms in order to transmit it securely.

[0440] A "server" is a computer system that receives and analyzes data sent from a terminal.

[0441] "Decryption" is the process of restoring encrypted data to its original state.

[0442] "AI technology" is a data analysis method that utilizes artificial intelligence, and is used to perform pattern recognition and anomaly detection.

[0443] A "database" is a system for storing analytical results and other information.

[0444] "Notification" is a means of sending information to care providers and family members when an abnormality is detected.

[0445] System configuration

[0446] This invention is a system for providing a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities.

[0447] Key Components

[0448] 1. Sensor group (terminal)

[0449] The device includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. These sensors monitor the environment in real time and collect data. For example, when a motion sensor detects movement, the information is collected immediately.

[0450] 2. Data transmission device (terminal)

[0451] The device sends the collected sensor data to a server at regular intervals. This process encrypts the data using an encryption algorithm such as AES (Advanced Encryption Standard). Specifically, security is ensured by sending encrypted data.

[0452] 3. Server

[0453] The server decrypts the received data and analyzes it using AI technology, such as Python machine learning libraries (e.g., TensorFlow). The server compares the decrypted data with past data to detect abnormal patterns and behavioral patterns.

[0454] 4. Notification System (Server)

[0455] If the server detects an abnormality, it will notify the care provider or family in real time via email, SMS, voice call, etc. The notification will include a detailed description of the type of abnormality and countermeasures.

[0456] Specific examples

[0457] For example, the system may be implemented in a home as follows: A motion sensor detects unusual activity in the hallway. Because this activity is unusual, the sensor collects data and the device sends it to a server. The server analyzes the received data and detects abnormal patterns. Based on the analysis, an abnormality is confirmed and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification includes the specific nature of the abnormality, the location information, and specific instructions for how to respond.

[0458] Example prompts for generative AI models

[0459] "Please describe a sensor system that provides an environment where elderly people can live safely. The system collects data from motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors, analyzes it on a server, and detects abnormalities. For example, please include how to handle abnormal activity in the hallway."

[0460] This invention will realize a home environment where the elderly and people with disabilities can live safely. The system enables early detection of abnormalities and rapid response, reducing the risk of emergencies.

[0461] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0462] Step 1:

[0463] Sensor data collection

[0464] The device collects data from motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors.

[0465] As input, real-time data from each sensor is provided, and the device temporarily stores this data in its memory.

[0466] As a specific example of operation, when a motion sensor detects movement in a hallway, the detected data is stored in memory.

[0467] As an output, all collected sensor data is stored in memory.

[0468] Step 2:

[0469] Data encryption and transmission

[0470] The terminal encrypts the collected sensor data using the AES encryption algorithm and sends it to the server.

[0471] The input is the sensor data collected in step 1.

[0472] An encryption process is performed to generate encrypted data, which is then sent from the terminal to the server.

[0473] As a specific example of operation, data from a motion sensor in a hallway is encrypted and sent to a server.

[0474] As output, the encrypted sensor data is sent to the server.

[0475] Step 3:

[0476] Receiving and Decrypting Data

[0477] The server receives the data sent from the terminal and decrypts the data through a decryption process.

[0478] As input, encrypted sensor data is provided.

[0479] A decoding process is performed to reconstruct the original sensor data.

[0480] As a specific example of operation, the server receives encrypted motion sensor data and decrypts it to obtain the original data.

[0481] As an output, the decoded sensor data is obtained.

[0482] Step 4:

[0483] Data analysis

[0484] The server analyzes the decoded sensor data using AI technology.

[0485] As input, the decoded sensor data is provided.

[0486] Data analysis involves comparing data with past data and using machine learning libraries (e.g., TensorFlow) to detect behavioral and anomalous patterns.

[0487] As a specific example of operation, the server analyzes data from a motion sensor and detects movements that differ from normal behavior patterns.

[0488] The output is the analysis result, which includes a judgment of normality or abnormality.

[0489] Step 5:

[0490] Anomaly detection

[0491] The server detects anomalies based on the results of data analysis.

[0492] As input, the analysis results obtained in step 4 are provided.

[0493] If an anomaly is detected, an anomaly flag is set in the analysis results.

[0494] As a specific example of operation, the server detects abnormal movement in a hallway and adds an abnormality flag.

[0495] The output is an anomaly detection result, which is an analysis result with an anomaly flag added.

[0496] Step 6:

[0497] Send notifications

[0498] If an abnormality is detected, the server sends a notification to the care provider or family.

[0499] As input, the analysis results in which an anomaly was detected are provided.

[0500] A notification will be generated and sent via email, SMS, voice call, etc. The notification will include the specific anomaly and its location.

[0501] As a specific example of operation, the server detects an abnormality and sends a notification of the abnormality to the care provider and family via email and SMS.

[0502] As an output, a notification is sent: a notification message containing the anomaly description and location information is generated and sent to the specified contacts.

[0503] (Application example 1)

[0504] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0505] In conventional residential environments, monitoring systems have been inadequate to ensure the safety of elderly and disabled people. Furthermore, there has been no system in brick-and-mortar stores that can detect and notify abnormalities in real time while ensuring the safety of elderly and disabled customers. The purpose of this invention is to solve these problems and provide an environment where elderly and disabled people can live and visit stores with peace of mind.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0507] In this invention, the server includes means for collecting data from a plurality of sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for transmitting a notification when the abnormality is detected, means for detecting environmental abnormalities with sensors installed in the store, and means for transmitting a notification to a smartphone when the environmental abnormality is detected. This not only enables elderly people and people with disabilities to live in peace of mind, but also makes it possible to ensure the safety of elderly and disabled customers in physical stores in real time.

[0508] A "sensor" is a device that acquires environmental information in real time and collects it as data.

[0509] The "data transmission means" is a means for transmitting data collected by the sensor to the server.

[0510] A "server" is a computing device that analyzes collected data and detects anomalies.

[0511] The "anomaly detection means" is a means for detecting unusual patterns or anomalies from the analyzed data.

[0512] The "notification sending means" is a means for sending a notification to relevant parties when an abnormality is detected.

[0513] A "smartphone" is a portable multi-function information terminal that has communication capabilities and can run applications.

[0514] "Inside the store" refers to the interior space of a store that sells goods.

[0515] "Environmental abnormalities" refers to situations that deviate from normal conditions, such as high temperatures, smoke, gas leaks, etc.

[0516] A "motion sensor" is a sensor for detecting movement.

[0517] This system provides a safe and secure environment for the elderly and people with disabilities, and has monitoring and notification functions using sensors, servers, and smartphones. Furthermore, this system can also be applied to similar safety systems in brick-and-mortar stores.

[0518] System configuration

[0519] Sensor group

[0520] The system includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors and window sensors, which are installed at different locations in brick-and-mortar stores and homes to monitor environmental information in real time.

[0521] Data transmission method

[0522] The data collected from the sensors is encrypted before being sent to the server, ensuring data security.

[0523] server

[0524] The server decodes the received sensor data and analyzes it using AI technology. If an anomaly is detected as a result of the analysis, a notification is triggered. The server also has the function of storing the anomaly detection results in a database.

[0525] Notification sending method

[0526] If an abnormality is detected, the server will immediately send email, SMS, or in-app notifications to relevant people (e.g., care providers and family members) via smartphone applications, ensuring the safety of seniors and people with disabilities in real time.

[0527] Specific examples

[0528] Consider a specific example of implementing the present invention in a brick-and-mortar store via a smartphone application called "Safety Navi." Sensors installed at various points in the store collect environmental information in real time, and if an abnormality is detected, the data is sent to a server.

[0529] If the server analyzes the data and detects an abnormality, the abnormality information is sent to the customer's smartphone via the app. The notification is displayed in the following format:

[0530] Prompt Sentence Examples

[0531] "A high temperature alarm has been detected near the refrigerated display case. We are instructing customers in the area to move away immediately."

[0532] This will enable elderly and disabled customers to safely visit and enjoy shopping in physical stores, and if a situation requires attention, immediate notification will enable a prompt response.

[0533] Hardware and Software Used

[0534] Sensors: Motion sensor, temperature sensor, smoke sensor, gas sensor, door sensor, window sensor

[0535] Server: Used for data analysis and anomaly detection

[0536] Smartphone: Used to receive notifications

[0537] Software: Data analysis and notification system implementation using Python, specifically using libraries such as requests, smtplib, and json.

[0538] This will make it possible to provide residential and retail environments where elderly people and people with disabilities can live safely.

[0539] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0540] Step 1:

[0541] Sensors collect real-time environmental information (motion, temperature, smoke, gas, door and window data) within physical stores and homes.

[0542] Input: Physical variables of the environment (motion, temperature, smoke, gases, open and closed doors and windows).

[0543] Data processing: Sensors convert physical variables into electrical signals, which are then generated as digital data.

[0544] Output: Digital data (environmental information).

[0545] Step 2:

[0546] The device sends the collected sensor data to the server at regular intervals, and the data is encrypted to maintain security.

[0547] Input: Sensor data (digital data).

[0548] Data Computation: The sensor data is encrypted and sent to the server using a secure channel.

[0549] Output: Encrypted sensor data.

[0550] Step 3:

[0551] The server decrypts the received encrypted data and analyzes the environmental data.

[0552] Input: Encrypted sensor data.

[0553] Data computation: Decrypting the data and using AI techniques to analyze it, for example, using generative AI models to detect abnormal patterns and compare them with normal behavioral patterns.

[0554] Output: Analysis results (anomaly detection results).

[0555] Step 4:

[0556] The server generates a notification if it detects an anomaly based on the analysis results.

[0557] Input: Analysis results.

[0558] Data calculation: Based on the analysis results, a notification message is generated if an abnormality is detected.

[0559] Output: Informational message.

[0560] Step 5:

[0561] The server sends the generated notification messages to the smartphones of care providers, family members, store managers, and customers.

[0562] Input: Notification message.

[0563] Data calculation: Send notification messages as email, SMS, or in-app notifications.

[0564] Output: The notification sent.

[0565] Step 6:

[0566] The user receives a notification message via their smartphone and takes necessary measures.

[0567] Input: Notification message.

[0568] Specific actions: The user checks the contents of the message and takes action according to the specific instructions, such as avoiding a specific location or contacting an administrator.

[0569] Output: User behavior (safety action).

[0570] This will allow the entire system to be integrated and provide a safe environment for seniors, people with disabilities, and brick-and-mortar store visitors.

[0571] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0572] This invention combines an emotion engine that recognizes the user's emotions with a system that provides a home environment where the elderly and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities. It also recognizes the user's emotional state and takes necessary measures.

[0573] System configuration

[0574] The system mainly consists of the following components:

[0575] 1. Sensors

[0576] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0577] 2. Data transmission device

[0578] The device sends the collected sensor data to a server at regular intervals, and this process is performed by encrypting the data for security reasons.

[0579] 3. Server

[0580] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0581] 4. Emotion Engine

[0582] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine uses image processing and voice analysis technologies to determine emotions from the user's facial expressions and vocalizations. This allows the server to monitor the user's mental state and notify care providers or family members as necessary.

[0583] 5. Notification System

[0584] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[0585] System Operation

[0586] The device first collects real-time data from multiple sensors placed in the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and the opening and closing status of doors and windows are continuously monitored.

[0587] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0588] The emotion engine analyzes the user's emotions from collected video and audio data. For example, it can determine whether the user is sad or angry from their facial expression, or recognize stress levels from their tone of voice.

[0589] If an abnormality or emotional change is detected, the server will immediately notify the care provider or family member in real time, enabling a rapid response. For example, if an elderly person collapses, a fire breaks out, or the user is experiencing extreme stress, a notification with detailed information and a prompt to take action will be sent.

[0590] Specific examples

[0591] For example, the system may be implemented in a home as follows: A motion sensor detects abnormal movement in the hallway. Furthermore, a facial recognition camera analyzes the user's facial expressions and detects a sad expression. This series of data is collected by the device, encrypted, and sent to a server. The server analyzes the data and detects abnormal movement and emotional changes. As a result, an abnormality is confirmed, and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification contains the specific details of the abnormality, the user's emotional state, and specific instructions for how to respond.

[0592] In this way, a system will be realized that provides an environment where elderly people and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, this system will also monitor the user's emotional state and contribute to maintaining mental health.

[0593] The processing flow will be explained below.

[0594] Step 1:

[0595] The device initializes multiple sensors (motion sensor, temperature sensor, smoke sensor, gas sensor, door sensor, window sensor) and checks their operation. Each sensor has a self-diagnostic function to check whether it is operating normally.

[0596] Step 2:

[0597] The device periodically collects data from each sensor. For example, a motion sensor acquires information whenever it detects movement. A temperature sensor measures the room temperature at regular intervals and updates the data. Smoke and gas sensors collect data immediately when they detect an abnormality.

[0598] Step 3:

[0599] The device converts the collected sensor data into JSON-formatted packets and encrypts them, preventing unauthorized access to the data during transmission.

[0600] Step 4:

[0601] The device sends encrypted sensor data to a server using a secure protocol such as HTTPS to protect the authenticity and privacy of the data.

[0602] Step 5:

[0603] The server decrypts the received data and converts it into JSON format data, which is then pre-processed for analysis.

[0604] Step 6:

[0605] The server performs anomaly detection on the received data, using an AI model (e.g., Isolation Forest) to determine whether the collected data is within normal ranges or if an abnormal pattern has been detected.

[0606] Step 7:

[0607] The server records the results of the anomaly detection in a database, and this data is saved as history for future reference.

[0608] Step 8:

[0609] The server uses image processing technology to analyze the user's facial expressions and recognize the user's emotions (e.g., joy, anger, sadness, etc.) based on the video data sent from the camera.

[0610] Step 9:

[0611] The server uses voice analysis technology to recognize emotions from the user's voice. Based on the audio data collected from the microphone, the server analyzes the tone and pitch of the user's voice to determine their emotional state.

[0612] Step 10:

[0613] The server combines the user's emotional data analyzed by the emotion engine with sensor data for a comprehensive analysis, which allows it to determine the user's overall health and safety.

[0614] Step 11:

[0615] The server notifies care providers and family members if an abnormality or emotional change is detected. The notification includes details of the detected abnormality, the user's emotional state, and recommended actions. Notifications are sent via email, SMS, or emergency call.

[0616] Step 12:

[0617] The server stores the analysis results and notification details in a database, which will be used for future analysis and report generation.

[0618] In this way, the system of the present invention specifically implements a series of processing flows including sensor data collection, anomaly detection, emotion recognition, notification, and data storage, thereby providing comprehensive support for the safety and mental health of users.

[0619] Example 2

[0620] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0621] To provide a residential environment where elderly people and people with disabilities can live safely, it is important to detect indoor environmental changes and abnormalities early and respond quickly. However, conventional systems only collect data and detect abnormalities using sensors, and are unable to grasp the user's emotional state, making it difficult to maintain mental health and provide appropriate care. The present invention aims to solve these problems and provide a residential environment where elderly people and people with disabilities can live more safely.

[0622] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0623] In this invention, the server includes means for collecting data from a plurality of sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for the server to analyze the user's emotions, and means for transmitting a notification when the server detects the abnormality or change in emotion. This enables early detection of abnormalities and rapid response, and maintenance of mental health by understanding the user's emotional state.

[0624] "Multiple sensors" refers to different types of sensors, such as motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.

[0625] "Means for collecting data" refers to the function of using multiple sensors to monitor the indoor environmental conditions and obtain data in real time.

[0626] "Means for transmitting data to a server" refers to the function for encrypting collected sensor data and transmitting it to a server via the Internet.

[0627] "Means for the server to analyze data and detect abnormalities" refers to the function in which the server uses AI technology to analyze the sensor data it receives and compare it with normal patterns to detect abnormalities early.

[0628] "Means for the server to analyze the user's emotions" refers to the function of determining the user's emotional state based on the video and audio data collected by the server using image processing technology and audio analysis technology.

[0629] "Means for sending notifications" refers to the function of notifying care providers and family members in real time via email, SMS, voice call, etc. when an abnormality or change in the user's emotions is detected.

[0630] This invention is a system for providing a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and can detect abnormalities and recognize the user's emotional state, taking necessary measures.

[0631] System configuration

[0632] The system consists of the following components:

[0633] 1. Sensors

[0634] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0635] 2. Data transmission device

[0636] The device sends the collected sensor data to a server at regular intervals. This process encrypts the data for security reasons. An encryption algorithm such as AES (Advanced Encryption Standard) is used.

[0637] 3. Server

[0638] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. AI models such as TensorFlow and PyTorch are used. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0639] 4. Emotion Engine

[0640] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine uses image processing technology (OpenCV, Dlib, etc.) and voice analysis technology (Google's voice recognition API, etc.) to determine emotions from the user's facial expressions and vocalizations.

[0641] 5. Notification System

[0642] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[0643] System Operation

[0644] The device first collects real-time data from multiple sensors placed in the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and the opening and closing status of doors and windows are continuously monitored.

[0645] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0646] The emotion engine analyzes the user's emotions from collected video and audio data. For example, it can determine whether the user is sad or angry from their facial expression, or recognize stress levels from their tone of voice.

[0647] If an abnormality or emotional change is detected, the server will immediately notify the care provider or family member in real time, enabling a rapid response. For example, if an elderly person collapses, a fire breaks out, or the user is experiencing extreme stress, a notification with detailed information and a prompt to take action will be sent.

[0648] Specific examples

[0649] For example, the system may be implemented in a home as follows: A motion sensor detects abnormal movement in the hallway. Furthermore, a facial recognition camera analyzes the user's facial expressions and detects a sad expression. This series of data is collected by the device, encrypted, and sent to a server. The server analyzes the data and detects abnormal movement and emotional changes. As a result, an abnormality is confirmed, and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification contains the specific details of the abnormality, the user's emotional state, and specific instructions for how to respond.

[0650] In this way, a system will be realized that provides an environment where elderly people and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, this system will also monitor the user's emotional state and contribute to maintaining mental health.

[0651] Prompt Sentence Examples

[0652] Below are some example prompts for input to a generative AI model:

[0653] Follow this prompt to explain in natural language the process of a system that provides a safe and secure living environment for the elderly and people with disabilities. This system uses multiple sensors to collect data and analyzes it on a server. It also uses an emotion engine to recognize the user's emotions and take necessary measures. Please provide specific steps and examples. Use the subject of either the server, device, or user, and use plain sentence endings.

[0654] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0655] Step 1:

[0656] The device collects data in real time from multiple sensors (motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.). The input data includes the motion, temperature, smoke, gas, and whether a door or window is open or closed, detected by the sensors. For example, when a motion sensor detects motion, that information is collected immediately and becomes input data. Specific operations include the process by which the device acquires data from the sensors. The output is the collected raw sensor data.

[0657] Step 2:

[0658] The terminal encrypts the collected sensor data. Here, an encryption algorithm such as AES (Advanced Encryption Standard) is used. The input is the raw sensor data collected in step 1. The specific operation includes the process of protecting the data using an encryption algorithm. The output is the encrypted sensor data.

[0659] Step 3:

[0660] The terminal sends the encrypted sensor data to the server via the Internet. The input is the sensor data encrypted in step 2. The specific operation includes a process of sending the data to the server using a communication protocol. The output is the encrypted data sent to the server.

[0661] Step 4:

[0662] The server receives and decrypts the encrypted data sent over the Internet. The input is the encrypted data sent in step 3. The specific operations include the process of decrypting the data using a decryption algorithm. The output is the decrypted sensor data.

[0663] Step 5:

[0664] The server analyzes the decoded sensor data and detects abnormal patterns. This is done using AI technology. The input is the sensor data decoded in step 4. Specific operations include analyzing the data using AI models such as TensorFlow and PyTorch to detect abnormalities. The output is the presence or absence of anomalies and the analysis results.

[0665] Step 6:

[0666] The server analyzes the user's emotions using the decoded video and audio data. The emotion engine uses image processing technologies (such as OpenCV and Dlib) and audio analysis technologies (such as Google's speech recognition API). The input is video and audio data. Specific operations include identifying the user's emotions from facial expressions and tone of voice. The output is the identified emotional state of the user.

[0667] Step 7:

[0668] The server notifies care providers and families if it detects anomalies or emotional changes. The inputs are the anomaly information detected in step 5 and the emotional state identified in step 6. Specific operations include sending information in real time via the notification system using email, SMS, or voice calls. The output is a notification sent to the care provider or family.

[0669] (Application example 2)

[0670] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] In order to provide an environment where the elderly and people with disabilities can live safely, it is important not only to detect abnormalities in the environment but also to monitor the user's emotional state and provide prompt care as needed. However, current systems are specialized in detecting abnormalities, and therefore have the problem of not being able to respond appropriately based on the user's emotional state.

[0672] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from multiple sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for transmitting a notification when the abnormality is detected, means for the server to analyze image data and voice data of the user to recognize emotions, and means for transmitting a notification according to the user's emotional state. This makes it possible to monitor the user's emotional state in addition to detecting abnormalities in the environment, and to respond quickly and appropriately.

[0673] The "means for collecting data from multiple sensors" is a device that acquires environmental data using various types of sensors such as motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors.

[0674] The "means for transmitting the data to the server" refers to a communication device and process for securely transmitting the collected sensor data to the server via the Internet or a dedicated line.

[0675] The "means by which the server analyzes the data and detects anomalies" refers to a system that uses artificial intelligence (AI) or machine learning algorithms to analyze data received on the server and detect unusual patterns or changes in the environment.

[0676] The "means for sending a notification when the abnormality is detected" refers to a system that, when the server confirms an abnormality, notifies pre-defined contacts (e.g., care providers or family members) of the details of the abnormality via email, SMS, voice call, etc.

[0677] "Means for recognizing emotions by analyzing user image and audio data" refers to artificial intelligence (AI) technology that analyzes video and audio data acquired through cameras and microphones, and determines the user's emotional state from their facial expressions and tone of voice.

[0678] The "means for sending notifications according to the user's emotional state" is a system that notifies care providers and family members in real time as needed based on the recognized emotions.

[0679] This invention is a system that provides a residential environment where elderly people and people with disabilities can live safely. Specifically, it collects data from multiple sensors, sends it to a server for analysis, and can detect abnormalities and recognize the user's emotional state. This system consists of the following main components:

[0680] 1. Sensors

[0681] The sensor group includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. These sensors are installed on the terminal to monitor the indoor environment in real time and collect data.

[0682] 2. Data transmission device

[0683] The device sends the collected sensor data to the server at regular intervals. The data is encrypted for security reasons.

[0684] 3. Server

[0685] The server analyzes the received sensor data. This analysis utilizes AI technology to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically saved in a database and have the function of sending prompt notifications in the event of an emergency.

[0686] 4. Emotion Engine

[0687] The server is equipped with an emotion engine that recognizes the user's emotions. This emotion engine uses image processing and voice analysis technologies to determine emotions from the user's facial expressions and voice, monitors the user's mental state, and notifies care providers or family members as necessary.

[0688] 5. Notification System

[0689] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[0690] As a concrete example of how this might work, the system might operate as follows:

[0691] In a home, when a motion sensor detects abnormal activity in the hallway, the data is collected by the device, encrypted, and sent to a server. A camera then captures the user's facial expressions, which an emotion engine analyzes to detect sadness. This data is then analyzed by the server, which identifies abnormalities and emotional changes. The server then sends an email and SMS notification to the care provider and family members. The notification describes the abnormality, the user's emotional state, and includes specific instructions for how to respond.

[0692] In this way, a system will be realized that provides an environment where the elderly and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, the system will also monitor the user's emotional state and contribute to maintaining mental health.

[0693] Example prompts for generative AI models

[0694] Read the code below and write a Python program that collects data from the camera and sensors, analyzes the user's emotions, and sends a notification if an abnormality occurs. Use OpenCV, requests, cryptography, and smtplib for the program.

[0695] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0696] Step 1:

[0697] The device collects real-time environmental data from multiple sensors (motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.). This input data includes measurements from each sensor (e.g., temperature, motion detection, gas concentration, etc.). The device temporarily stores the collected data and prepares it for the next step.

[0698] Step 2:

[0699] The device sends the collected sensor data to the server at regular intervals. During this process, the device encrypts the data using a secure communication protocol (e.g., TLS). The data includes measurements from each sensor, and the server receives and analyzes it.

[0700] Step 3:

[0701] The server analyzes the received sensor data using AI technology and machine learning algorithms. This analysis process identifies patterns in the data and compares them with normal behavior and environmental conditions to detect anomalies. The analysis results include whether an anomaly occurred and the type of anomaly detected.

[0702] Step 4:

[0703] The server analyzes the user's image and audio data collected from the camera and microphone to recognize emotions. The emotion engine uses image processing and audio analysis technologies to determine the user's emotional state (e.g., sadness, anger, happiness, etc.) from their facial expressions and tone of voice. The input for this process is video and audio data, and the output is the recognized emotional state.

[0704] Step 5:

[0705] The server continuously monitors the user's emotional state and detects abnormalities based on the analysis results. If an abnormality or a specific emotional state is detected, the server sends a notification to pre-defined care providers or family members. The notification includes the details of the abnormality, the user's emotional state, and countermeasures. The notification can be sent via email, SMS, voice call, or other methods.

[0706] Step 6:

[0707] The server stores the analysis results and notification history in a database. The stored data includes measurements from each sensor, the recognized emotional state, and the content of the sent notification. This allows the system to track the user's long-term behavioral patterns and emotional state and use them for future analysis and countermeasures.

[0708] Step 7:

[0709] Users can change system settings and update notification destinations as needed, allowing the system to respond flexibly to individual user needs. User input to change settings is sent to the server and reflected in the overall system operation.

[0710] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0711] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0712] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0713] [Third embodiment]

[0714] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0715] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0716] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0717] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0718] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0719] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0720] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0721] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0722] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0723] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0724] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0725] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0726] This invention is a system that provides a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities.

[0727] System configuration

[0728] The system mainly consists of the following components:

[0729] 1. Sensors

[0730] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0731] 2. Data transmission device

[0732] The device sends the collected sensor data to a server at regular intervals, and this process is performed by encrypting the data for security reasons.

[0733] 3. Server

[0734] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0735] 4. Notification System

[0736] When the server detects an abnormality, it notifies the care provider or family member via email, SMS, voice call, etc. The notification will include the type of abnormality and recommended measures.

[0737] System Operation

[0738] The device first collects real-time data from multiple sensors placed throughout the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and door and window opening and closing status are continuously monitored.

[0739] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0740] If an abnormality is detected, the server immediately notifies care providers and family members in real time, enabling them to respond quickly to emergencies. For example, if an elderly person collapses or a fire breaks out, a notification with detailed information and a prompt to take action is sent.

[0741] Specific examples

[0742] For example, the system may be implemented in a home as follows: A motion sensor detects unusual activity in the hallway. Because this activity is unusual, the sensor collects data and the device sends it to a server. The server analyzes the received data and detects abnormal patterns. Based on the analysis, an abnormality is confirmed and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification includes the specific nature of the abnormality, the location information, and specific instructions for how to respond.

[0743] In this way, a system will be realized that provides an environment where the elderly and people with disabilities can live with peace of mind. This system will enable early detection of abnormalities and prompt response, reducing risks in emergencies.

[0744] The processing flow will be explained below.

[0745] Step 1:

[0746] The device initializes multiple sensors and checks that they are operating normally. This includes using the self-diagnostic function of each sensor and reading the initial values ​​to check for abnormalities.

[0747] Step 2:

[0748] The device periodically collects data from each sensor, such as a motion sensor that detects the presence or absence of movement, a temperature sensor that measures the indoor temperature, a smoke sensor that detects smoke, a gas sensor that detects gas leaks, and sensors that monitor the opening and closing of doors and windows, and obtains data in real time.

[0749] Step 3:

[0750] The device packages the collected sensor data at regular intervals and converts it into a format that the server can understand, such as JSON. The data is encrypted during this process.

[0751] Step 4:

[0752] The device then sends the packaged and encrypted data to the server, where the communication is performed using a secure protocol (e.g., HTTPS) to ensure the data is secure.

[0753] Step 5:

[0754] The server receives the data sent from the device and decrypts it, converting it into a format suitable for analysis.

[0755] Step 6:

[0756] The server preprocesses the received data, checking for missing values ​​and outliers, and imputes data as needed.

[0757] Step 7:

[0758] The server then runs AI analysis on the pre-processed data, using AI models (such as Isolation Forest) to detect anomalies and determine whether they match a specific pattern.

[0759] Step 8:

[0760] If an anomaly is detected, the server determines the nature of the anomaly and generates the necessary notification, which includes the type of anomaly, its location, and recommended actions to take.

[0761] Step 9:

[0762] The server then sends notifications to care providers and family members via email, SMS, or voice calls.

[0763] Step 10:

[0764] The server stores the analysis results and sensor data in a database, which can then be used for later analysis and report generation.

[0765] Step 11:

[0766] The server periodically generates and provides users with reports that include anomaly detection history, behavioral pattern analysis results, and daily health trends.

[0767] In this way, the system of the present invention achieves a series of processing flows from collecting sensor data to detecting anomalies, notifying, storing data, and generating reports.

[0768] Example 1

[0769] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0770] Previously, systems designed to provide a safe and secure residential environment for the elderly and people with disabilities lacked the technology to detect abnormalities, making it difficult to detect them in real time. Furthermore, security issues existed regarding the safe transmission and storage of collected sensor data. This made it difficult to respond quickly and reliably to emergencies, and user safety was not adequately ensured.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0772] In this invention, the server includes means for receiving data from the terminal at regular intervals and encrypting the data before transmission, means for decrypting the received data, and means for analyzing the decrypted data using AI technology to detect anomalies, thereby enabling real-time anomaly detection and safe transmission and analysis of collected sensor data.

[0773] A "sensor" is a device that monitors environmental conditions and collects data.

[0774] A "motion sensor" is a sensor that detects the movement of an object and collects that information.

[0775] A "temperature sensor" is a sensor that measures the ambient temperature and collects that data.

[0776] A "smoke sensor" is a sensor that detects the presence of smoke and collects information about it.

[0777] A "gas sensor" is a sensor that detects the presence of harmful gases and collects information about them.

[0778] A "door sensor" is a sensor that detects whether a door is open or closed and collects that data.

[0779] A "window sensor" is a sensor that detects whether a window is open or closed and collects that data.

[0780] A "terminal" is a device that collects data from multiple sensors and transmits it to a server.

[0781] "Encryption" is the process of transforming data using special algorithms in order to transmit it securely.

[0782] A "server" is a computer system that receives and analyzes data sent from a terminal.

[0783] "Decryption" is the process of restoring encrypted data to its original state.

[0784] "AI technology" is a data analysis method that utilizes artificial intelligence, and is used to perform pattern recognition and anomaly detection.

[0785] A "database" is a system for storing analytical results and other information.

[0786] "Notification" is a means of sending information to care providers and family members when an abnormality is detected.

[0787] System configuration

[0788] This invention is a system for providing a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities.

[0789] Key Components

[0790] 1. Sensor group (terminal)

[0791] The device includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. These sensors monitor the environment in real time and collect data. For example, when a motion sensor detects movement, the information is collected immediately.

[0792] 2. Data transmission device (terminal)

[0793] The device sends the collected sensor data to a server at regular intervals. This process encrypts the data using an encryption algorithm such as AES (Advanced Encryption Standard). Specifically, security is ensured by sending encrypted data.

[0794] 3. Server

[0795] The server decrypts the received data and analyzes it using AI technology, such as Python machine learning libraries (e.g., TensorFlow). The server compares the decrypted data with past data to detect abnormal patterns and behavioral patterns.

[0796] 4. Notification System (Server)

[0797] If the server detects an abnormality, it will notify the care provider or family in real time via email, SMS, voice call, etc. The notification will include a detailed description of the type of abnormality and countermeasures.

[0798] Specific examples

[0799] For example, the system may be implemented in a home as follows: A motion sensor detects unusual activity in the hallway. Because this activity is unusual, the sensor collects data and the device sends it to a server. The server analyzes the received data and detects abnormal patterns. Based on the analysis, an abnormality is confirmed and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification includes the specific nature of the abnormality, the location information, and specific instructions for how to respond.

[0800] Example prompts for generative AI models

[0801] "Please describe a sensor system that provides an environment where elderly people can live safely. The system collects data from motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors, analyzes it on a server, and detects abnormalities. For example, please include how to handle abnormal activity in the hallway."

[0802] This invention will realize a home environment where the elderly and people with disabilities can live safely. The system enables early detection of abnormalities and rapid response, reducing the risk of emergencies.

[0803] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0804] Step 1:

[0805] Sensor data collection

[0806] The device collects data from motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors.

[0807] As input, real-time data from each sensor is provided, and the device temporarily stores this data in its memory.

[0808] As a specific example of operation, when a motion sensor detects movement in a hallway, the detected data is stored in memory.

[0809] As an output, all collected sensor data is stored in memory.

[0810] Step 2:

[0811] Data encryption and transmission

[0812] The terminal encrypts the collected sensor data using the AES encryption algorithm and sends it to the server.

[0813] The input is the sensor data collected in step 1.

[0814] An encryption process is performed to generate encrypted data, which is then sent from the terminal to the server.

[0815] As a specific example of operation, data from a motion sensor in a hallway is encrypted and sent to a server.

[0816] As output, the encrypted sensor data is sent to the server.

[0817] Step 3:

[0818] Receiving and Decrypting Data

[0819] The server receives the data sent from the terminal and decrypts the data through a decryption process.

[0820] As input, encrypted sensor data is provided.

[0821] A decoding process is performed to reconstruct the original sensor data.

[0822] As a specific example of operation, the server receives encrypted motion sensor data and decrypts it to obtain the original data.

[0823] As an output, the decoded sensor data is obtained.

[0824] Step 4:

[0825] Data analysis

[0826] The server analyzes the decoded sensor data using AI technology.

[0827] As input, the decoded sensor data is provided.

[0828] Data analysis involves comparing data with past data and using machine learning libraries (e.g., TensorFlow) to detect behavioral and anomalous patterns.

[0829] As a specific example of operation, the server analyzes data from a motion sensor and detects movements that differ from normal behavior patterns.

[0830] The output is the analysis result, which includes a judgment of normality or abnormality.

[0831] Step 5:

[0832] Anomaly detection

[0833] The server detects anomalies based on the results of data analysis.

[0834] As input, the analysis results obtained in step 4 are provided.

[0835] If an anomaly is detected, an anomaly flag is set in the analysis results.

[0836] As a specific example of operation, the server detects abnormal movement in a hallway and adds an abnormality flag.

[0837] The output is an anomaly detection result, which is an analysis result with an anomaly flag added.

[0838] Step 6:

[0839] Send notifications

[0840] If an abnormality is detected, the server sends a notification to the care provider or family.

[0841] As input, the analysis results in which an anomaly was detected are provided.

[0842] A notification will be generated and sent via email, SMS, voice call, etc. The notification will include the specific anomaly and its location.

[0843] As a specific example of operation, the server detects an abnormality and sends a notification of the abnormality to the care provider and family via email and SMS.

[0844] As an output, a notification is sent: a notification message containing the anomaly description and location information is generated and sent to the specified contacts.

[0845] (Application example 1)

[0846] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0847] In conventional residential environments, monitoring systems have been inadequate to ensure the safety of elderly and disabled people. Furthermore, there has been no system in brick-and-mortar stores that can detect and notify abnormalities in real time while ensuring the safety of elderly and disabled customers. The purpose of this invention is to solve these problems and provide an environment where elderly and disabled people can live and visit stores with peace of mind.

[0848] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0849] In this invention, the server includes means for collecting data from a plurality of sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for transmitting a notification when the abnormality is detected, means for detecting environmental abnormalities with sensors installed in the store, and means for transmitting a notification to a smartphone when the environmental abnormality is detected. This not only enables elderly people and people with disabilities to live in peace of mind, but also makes it possible to ensure the safety of elderly and disabled customers in physical stores in real time.

[0850] A "sensor" is a device that acquires environmental information in real time and collects it as data.

[0851] The "data transmission means" is a means for transmitting data collected by the sensor to the server.

[0852] A "server" is a computing device that analyzes collected data and detects anomalies.

[0853] The "anomaly detection means" is a means for detecting unusual patterns or anomalies from the analyzed data.

[0854] The "notification sending means" is a means for sending a notification to relevant parties when an abnormality is detected.

[0855] A "smartphone" is a portable multi-function information terminal that has communication capabilities and can run applications.

[0856] "Inside the store" refers to the interior space of a store that sells goods.

[0857] "Environmental abnormalities" refers to situations that deviate from normal conditions, such as high temperatures, smoke, gas leaks, etc.

[0858] A "motion sensor" is a sensor for detecting movement.

[0859] This system provides a safe and secure environment for the elderly and people with disabilities, and has monitoring and notification functions using sensors, servers, and smartphones. Furthermore, this system can also be applied to similar safety systems in brick-and-mortar stores.

[0860] System configuration

[0861] Sensor group

[0862] The system includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors and window sensors, which are installed at different locations in brick-and-mortar stores and homes to monitor environmental information in real time.

[0863] Data transmission method

[0864] The data collected from the sensors is encrypted before being sent to the server, ensuring data security.

[0865] server

[0866] The server decodes the received sensor data and analyzes it using AI technology. If an anomaly is detected as a result of the analysis, a notification is triggered. The server also has the function of storing the anomaly detection results in a database.

[0867] Notification sending method

[0868] If an abnormality is detected, the server will immediately send email, SMS, or in-app notifications to relevant people (e.g., care providers and family members) via smartphone applications, ensuring the safety of seniors and people with disabilities in real time.

[0869] Specific examples

[0870] Consider a specific example of implementing the present invention in a brick-and-mortar store via a smartphone application called "Safety Navi." Sensors installed at various points in the store collect environmental information in real time, and if an abnormality is detected, the data is sent to a server.

[0871] If the server analyzes the data and detects an abnormality, the abnormality information is sent to the customer's smartphone via the app. The notification is displayed in the following format:

[0872] Prompt Sentence Examples

[0873] "A high temperature alarm has been detected near the refrigerated display case. We are instructing customers in the area to move away immediately."

[0874] This will enable elderly and disabled customers to safely visit and enjoy shopping in physical stores, and if a situation requires attention, immediate notification will enable a prompt response.

[0875] Hardware and Software Used

[0876] Sensors: Motion sensor, temperature sensor, smoke sensor, gas sensor, door sensor, window sensor

[0877] Server: Used for data analysis and anomaly detection

[0878] Smartphone: Used to receive notifications

[0879] Software: Data analysis and notification system implementation using Python, specifically using libraries such as requests, smtplib, and json.

[0880] This will make it possible to provide residential and retail environments where elderly people and people with disabilities can live safely.

[0881] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0882] Step 1:

[0883] Sensors collect real-time environmental information (motion, temperature, smoke, gas, door and window data) within physical stores and homes.

[0884] Input: Physical variables of the environment (motion, temperature, smoke, gases, open and closed doors and windows).

[0885] Data processing: Sensors convert physical variables into electrical signals, which are then generated as digital data.

[0886] Output: Digital data (environmental information).

[0887] Step 2:

[0888] The device sends the collected sensor data to the server at regular intervals, and the data is encrypted to maintain security.

[0889] Input: Sensor data (digital data).

[0890] Data Computation: The sensor data is encrypted and sent to the server using a secure channel.

[0891] Output: Encrypted sensor data.

[0892] Step 3:

[0893] The server decrypts the received encrypted data and analyzes the environmental data.

[0894] Input: Encrypted sensor data.

[0895] Data computation: Decrypting the data and using AI techniques to analyze it, for example, using generative AI models to detect abnormal patterns and compare them with normal behavioral patterns.

[0896] Output: Analysis results (anomaly detection results).

[0897] Step 4:

[0898] The server generates a notification if it detects an anomaly based on the analysis results.

[0899] Input: Analysis results.

[0900] Data calculation: Based on the analysis results, a notification message is generated if an abnormality is detected.

[0901] Output: Informational message.

[0902] Step 5:

[0903] The server sends the generated notification messages to the smartphones of care providers, family members, store managers, and customers.

[0904] Input: Notification message.

[0905] Data calculation: Send notification messages as email, SMS, or in-app notifications.

[0906] Output: The notification sent.

[0907] Step 6:

[0908] The user receives a notification message via their smartphone and takes necessary measures.

[0909] Input: Notification message.

[0910] Specific actions: The user checks the contents of the message and takes action according to the specific instructions, such as avoiding a specific location or contacting an administrator.

[0911] Output: User behavior (safety action).

[0912] This will allow the entire system to be integrated and provide a safe environment for seniors, people with disabilities, and brick-and-mortar store visitors.

[0913] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0914] This invention combines an emotion engine that recognizes the user's emotions with a system that provides a home environment where the elderly and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities. It also recognizes the user's emotional state and takes necessary measures.

[0915] System configuration

[0916] The system mainly consists of the following components:

[0917] 1. Sensors

[0918] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0919] 2. Data transmission device

[0920] The device sends the collected sensor data to a server at regular intervals, and this process is performed by encrypting the data for security reasons.

[0921] 3. Server

[0922] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0923] 4. Emotion Engine

[0924] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine uses image processing and voice analysis technologies to determine emotions from the user's facial expressions and vocalizations. This allows the server to monitor the user's mental state and notify care providers or family members as necessary.

[0925] 5. Notification System

[0926] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[0927] System Operation

[0928] The device first collects real-time data from multiple sensors placed in the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and the opening and closing status of doors and windows are continuously monitored.

[0929] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0930] The emotion engine analyzes the user's emotions from collected video and audio data. For example, it can determine whether the user is sad or angry from their facial expression, or recognize stress levels from their tone of voice.

[0931] If an abnormality or emotional change is detected, the server will immediately notify the care provider or family member in real time, enabling a rapid response. For example, if an elderly person collapses, a fire breaks out, or the user is experiencing extreme stress, a notification with detailed information and a prompt to take action will be sent.

[0932] Specific examples

[0933] For example, the system may be implemented in a home as follows: A motion sensor detects abnormal movement in the hallway. Furthermore, a facial recognition camera analyzes the user's facial expressions and detects a sad expression. This series of data is collected by the device, encrypted, and sent to a server. The server analyzes the data and detects abnormal movement and emotional changes. As a result, an abnormality is confirmed, and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification contains the specific details of the abnormality, the user's emotional state, and specific instructions for how to respond.

[0934] In this way, a system will be realized that provides an environment where elderly people and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, this system will also monitor the user's emotional state and contribute to maintaining mental health.

[0935] The processing flow will be explained below.

[0936] Step 1:

[0937] The device initializes multiple sensors (motion sensor, temperature sensor, smoke sensor, gas sensor, door sensor, window sensor) and checks their operation. Each sensor has a self-diagnostic function to check whether it is operating normally.

[0938] Step 2:

[0939] The device periodically collects data from each sensor. For example, a motion sensor acquires information whenever it detects movement. A temperature sensor measures the room temperature at regular intervals and updates the data. Smoke and gas sensors collect data immediately when they detect an abnormality.

[0940] Step 3:

[0941] The device converts the collected sensor data into JSON-formatted packets and encrypts them, preventing unauthorized access to the data during transmission.

[0942] Step 4:

[0943] The device sends encrypted sensor data to a server using a secure protocol such as HTTPS to protect the authenticity and privacy of the data.

[0944] Step 5:

[0945] The server decrypts the received data and converts it into JSON format data, which is then pre-processed for analysis.

[0946] Step 6:

[0947] The server performs anomaly detection on the received data, using an AI model (e.g., Isolation Forest) to determine whether the collected data is within normal ranges or if an abnormal pattern has been detected.

[0948] Step 7:

[0949] The server records the results of the anomaly detection in a database, and this data is saved as history for future reference.

[0950] Step 8:

[0951] The server uses image processing technology to analyze the user's facial expressions and recognize the user's emotions (e.g., joy, anger, sadness, etc.) based on the video data sent from the camera.

[0952] Step 9:

[0953] The server uses voice analysis technology to recognize emotions from the user's voice. Based on the audio data collected from the microphone, the server analyzes the tone and pitch of the user's voice to determine their emotional state.

[0954] Step 10:

[0955] The server combines the user's emotional data analyzed by the emotion engine with sensor data for a comprehensive analysis, which allows it to determine the user's overall health and safety.

[0956] Step 11:

[0957] The server notifies care providers and family members if an abnormality or emotional change is detected. The notification includes details of the detected abnormality, the user's emotional state, and recommended actions. Notifications are sent via email, SMS, or emergency call.

[0958] Step 12:

[0959] The server stores the analysis results and notification details in a database, which will be used for future analysis and report generation.

[0960] In this way, the system of the present invention specifically implements a series of processing flows including sensor data collection, anomaly detection, emotion recognition, notification, and data storage, thereby providing comprehensive support for the safety and mental health of users.

[0961] Example 2

[0962] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0963] To provide a residential environment where elderly people and people with disabilities can live safely, it is important to detect indoor environmental changes and abnormalities early and respond quickly. However, conventional systems only collect data and detect abnormalities using sensors, and are unable to grasp the user's emotional state, making it difficult to maintain mental health and provide appropriate care. The present invention aims to solve these problems and provide a residential environment where elderly people and people with disabilities can live more safely.

[0964] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0965] In this invention, the server includes means for collecting data from a plurality of sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for the server to analyze the user's emotions, and means for transmitting a notification when the server detects the abnormality or change in emotion. This enables early detection of abnormalities and rapid response, and maintenance of mental health by understanding the user's emotional state.

[0966] "Multiple sensors" refers to different types of sensors, such as motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.

[0967] "Means for collecting data" refers to the function of using multiple sensors to monitor the indoor environmental conditions and obtain data in real time.

[0968] "Means for transmitting data to a server" refers to the function for encrypting collected sensor data and transmitting it to a server via the Internet.

[0969] "Means for the server to analyze data and detect abnormalities" refers to the function in which the server uses AI technology to analyze the sensor data it receives and compare it with normal patterns to detect abnormalities early.

[0970] "Means for the server to analyze the user's emotions" refers to the function of determining the user's emotional state based on the video and audio data collected by the server using image processing technology and audio analysis technology.

[0971] "Means for sending notifications" refers to the function of notifying care providers and family members in real time via email, SMS, voice call, etc. when an abnormality or change in the user's emotions is detected.

[0972] This invention is a system for providing a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and can detect abnormalities and recognize the user's emotional state, taking necessary measures.

[0973] System configuration

[0974] The system consists of the following components:

[0975] 1. Sensors

[0976] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[0977] 2. Data transmission device

[0978] The device sends the collected sensor data to a server at regular intervals. This process encrypts the data for security reasons. An encryption algorithm such as AES (Advanced Encryption Standard) is used.

[0979] 3. Server

[0980] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. AI models such as TensorFlow and PyTorch are used. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[0981] 4. Emotion Engine

[0982] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine uses image processing technology (OpenCV, Dlib, etc.) and voice analysis technology (Google's voice recognition API, etc.) to determine emotions from the user's facial expressions and vocalizations.

[0983] 5. Notification System

[0984] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[0985] System Operation

[0986] The device first collects real-time data from multiple sensors placed in the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and the opening and closing status of doors and windows are continuously monitored.

[0987] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[0988] The emotion engine analyzes the user's emotions from collected video and audio data. For example, it can determine whether the user is sad or angry from their facial expression, or recognize stress levels from their tone of voice.

[0989] If an abnormality or emotional change is detected, the server will immediately notify the care provider or family member in real time, enabling a rapid response. For example, if an elderly person collapses, a fire breaks out, or the user is experiencing extreme stress, a notification with detailed information and a prompt to take action will be sent.

[0990] Specific examples

[0991] For example, the system may be implemented in a home as follows: A motion sensor detects abnormal movement in the hallway. Furthermore, a facial recognition camera analyzes the user's facial expressions and detects a sad expression. This series of data is collected by the device, encrypted, and sent to a server. The server analyzes the data and detects abnormal movement and emotional changes. As a result, an abnormality is confirmed, and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification contains the specific details of the abnormality, the user's emotional state, and specific instructions for how to respond.

[0992] In this way, a system will be realized that provides an environment where elderly people and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, this system will also monitor the user's emotional state and contribute to maintaining mental health.

[0993] Prompt Sentence Examples

[0994] Below are some example prompts for input to a generative AI model:

[0995] Follow this prompt to explain in natural language the process of a system that provides a safe and secure living environment for the elderly and people with disabilities. This system uses multiple sensors to collect data and analyzes it on a server. It also uses an emotion engine to recognize the user's emotions and take necessary measures. Please provide specific steps and examples. Use the subject of either the server, device, or user, and use plain sentence endings.

[0996] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0997] Step 1:

[0998] The device collects data in real time from multiple sensors (motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.). The input data includes the motion, temperature, smoke, gas, and whether a door or window is open or closed, detected by the sensors. For example, when a motion sensor detects motion, that information is collected immediately and becomes input data. Specific operations include the process by which the device acquires data from the sensors. The output is the collected raw sensor data.

[0999] Step 2:

[1000] The terminal encrypts the collected sensor data. Here, an encryption algorithm such as AES (Advanced Encryption Standard) is used. The input is the raw sensor data collected in step 1. The specific operation includes the process of protecting the data using an encryption algorithm. The output is the encrypted sensor data.

[1001] Step 3:

[1002] The terminal sends the encrypted sensor data to the server via the Internet. The input is the sensor data encrypted in step 2. The specific operation includes a process of sending the data to the server using a communication protocol. The output is the encrypted data sent to the server.

[1003] Step 4:

[1004] The server receives and decrypts the encrypted data sent over the Internet. The input is the encrypted data sent in step 3. The specific operations include the process of decrypting the data using a decryption algorithm. The output is the decrypted sensor data.

[1005] Step 5:

[1006] The server analyzes the decoded sensor data and detects abnormal patterns. This is done using AI technology. The input is the sensor data decoded in step 4. Specific operations include analyzing the data using AI models such as TensorFlow and PyTorch to detect abnormalities. The output is the presence or absence of anomalies and the analysis results.

[1007] Step 6:

[1008] The server analyzes the user's emotions using the decoded video and audio data. The emotion engine uses image processing technologies (such as OpenCV and Dlib) and audio analysis technologies (such as Google's speech recognition API). The input is video and audio data. Specific operations include identifying the user's emotions from facial expressions and tone of voice. The output is the identified emotional state of the user.

[1009] Step 7:

[1010] The server notifies care providers and families if it detects anomalies or emotional changes. The inputs are the anomaly information detected in step 5 and the emotional state identified in step 6. Specific operations include sending information in real time via the notification system using email, SMS, or voice calls. The output is a notification sent to the care provider or family.

[1011] (Application example 2)

[1012] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1013] In order to provide an environment where the elderly and people with disabilities can live safely, it is important not only to detect abnormalities in the environment but also to monitor the user's emotional state and provide prompt care as needed. However, current systems are specialized in detecting abnormalities, and therefore have the problem of not being able to respond appropriately based on the user's emotional state.

[1014] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from multiple sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for transmitting a notification when the abnormality is detected, means for the server to analyze image data and voice data of the user to recognize emotions, and means for transmitting a notification according to the user's emotional state. This makes it possible to monitor the user's emotional state in addition to detecting abnormalities in the environment, and to respond quickly and appropriately.

[1015] The "means for collecting data from multiple sensors" is a device that acquires environmental data using various types of sensors such as motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors.

[1016] The "means for transmitting the data to the server" refers to a communication device and process for securely transmitting the collected sensor data to the server via the Internet or a dedicated line.

[1017] The "means by which the server analyzes the data and detects anomalies" refers to a system that uses artificial intelligence (AI) or machine learning algorithms to analyze data received on the server and detect unusual patterns or changes in the environment.

[1018] The "means for sending a notification when the abnormality is detected" refers to a system that, when the server confirms an abnormality, notifies pre-defined contacts (e.g., care providers or family members) of the details of the abnormality via email, SMS, voice call, etc.

[1019] "Means for recognizing emotions by analyzing user image and audio data" refers to artificial intelligence (AI) technology that analyzes video and audio data acquired through cameras and microphones, and determines the user's emotional state from their facial expressions and tone of voice.

[1020] The "means for sending notifications according to the user's emotional state" is a system that notifies care providers and family members in real time as needed based on the recognized emotions.

[1021] This invention is a system that provides a residential environment where elderly people and people with disabilities can live safely. Specifically, it collects data from multiple sensors, sends it to a server for analysis, and can detect abnormalities and recognize the user's emotional state. This system consists of the following main components:

[1022] 1. Sensors

[1023] The sensor group includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. These sensors are installed on the terminal to monitor the indoor environment in real time and collect data.

[1024] 2. Data transmission device

[1025] The device sends the collected sensor data to the server at regular intervals. The data is encrypted for security reasons.

[1026] 3. Server

[1027] The server analyzes the received sensor data. This analysis utilizes AI technology to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically saved in a database and have the function of sending prompt notifications in the event of an emergency.

[1028] 4. Emotion Engine

[1029] The server is equipped with an emotion engine that recognizes the user's emotions. This emotion engine uses image processing and voice analysis technologies to determine emotions from the user's facial expressions and voice, monitors the user's mental state, and notifies care providers or family members as necessary.

[1030] 5. Notification System

[1031] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[1032] As a concrete example of how this might work, the system might operate as follows:

[1033] In a home, when a motion sensor detects abnormal activity in the hallway, the data is collected by the device, encrypted, and sent to a server. A camera then captures the user's facial expressions, which an emotion engine analyzes to detect sadness. This data is then analyzed by the server, which identifies abnormalities and emotional changes. The server then sends an email and SMS notification to the care provider and family members. The notification describes the abnormality, the user's emotional state, and includes specific instructions for how to respond.

[1034] In this way, a system will be realized that provides an environment where the elderly and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, the system will also monitor the user's emotional state and contribute to maintaining mental health.

[1035] Example prompts for generative AI models

[1036] Read the code below and write a Python program that collects data from the camera and sensors, analyzes the user's emotions, and sends a notification if an abnormality occurs. Use OpenCV, requests, cryptography, and smtplib for the program.

[1037] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1038] Step 1:

[1039] The device collects real-time environmental data from multiple sensors (motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.). This input data includes measurements from each sensor (e.g., temperature, motion detection, gas concentration, etc.). The device temporarily stores the collected data and prepares it for the next step.

[1040] Step 2:

[1041] The device sends the collected sensor data to the server at regular intervals. During this process, the device encrypts the data using a secure communication protocol (e.g., TLS). The data includes measurements from each sensor, and the server receives and analyzes it.

[1042] Step 3:

[1043] The server analyzes the received sensor data using AI technology and machine learning algorithms. This analysis process identifies patterns in the data and compares them with normal behavior and environmental conditions to detect anomalies. The analysis results include whether an anomaly occurred and the type of anomaly detected.

[1044] Step 4:

[1045] The server analyzes the user's image and audio data collected from the camera and microphone to recognize emotions. The emotion engine uses image processing and audio analysis technologies to determine the user's emotional state (e.g., sadness, anger, happiness, etc.) from their facial expressions and tone of voice. The input for this process is video and audio data, and the output is the recognized emotional state.

[1046] Step 5:

[1047] The server continuously monitors the user's emotional state and detects abnormalities based on the analysis results. If an abnormality or a specific emotional state is detected, the server sends a notification to pre-defined care providers or family members. The notification includes the details of the abnormality, the user's emotional state, and countermeasures. The notification can be sent via email, SMS, voice call, or other methods.

[1048] Step 6:

[1049] The server stores the analysis results and notification history in a database. The stored data includes measurements from each sensor, the recognized emotional state, and the content of the sent notification. This allows the system to track the user's long-term behavioral patterns and emotional state and use them for future analysis and countermeasures.

[1050] Step 7:

[1051] Users can change system settings and update notification destinations as needed, allowing the system to respond flexibly to individual user needs. User input to change settings is sent to the server and reflected in the overall system operation.

[1052] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1053] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1054] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1055] [Fourth embodiment]

[1056] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1057] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1058] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1059] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1060] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1061] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1062] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1063] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1064] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1065] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1066] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1067] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1068] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1069] This invention is a system that provides a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities.

[1070] System configuration

[1071] The system mainly consists of the following components:

[1072] 1. Sensors

[1073] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[1074] 2. Data transmission device

[1075] The device sends the collected sensor data to a server at regular intervals, and this process is performed by encrypting the data for security reasons.

[1076] 3. Server

[1077] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[1078] 4. Notification System

[1079] When the server detects an abnormality, it notifies the care provider or family member via email, SMS, voice call, etc. The notification will include the type of abnormality and recommended measures.

[1080] System Operation

[1081] The device first collects real-time data from multiple sensors placed throughout the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and door and window opening and closing status are continuously monitored.

[1082] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[1083] If an abnormality is detected, the server immediately notifies care providers and family members in real time, enabling them to respond quickly to emergencies. For example, if an elderly person collapses or a fire breaks out, a notification with detailed information and a prompt to take action is sent.

[1084] Specific examples

[1085] For example, the system may be implemented in a home as follows: A motion sensor detects unusual activity in the hallway. Because this activity is unusual, the sensor collects data and the device sends it to a server. The server analyzes the received data and detects abnormal patterns. Based on the analysis, an abnormality is confirmed and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification includes the specific nature of the abnormality, the location information, and specific instructions for how to respond.

[1086] In this way, a system will be realized that provides an environment where the elderly and people with disabilities can live with peace of mind. This system will enable early detection of abnormalities and prompt response, reducing risks in emergencies.

[1087] The processing flow will be explained below.

[1088] Step 1:

[1089] The device initializes multiple sensors and checks that they are operating normally. This includes using the self-diagnostic function of each sensor and reading the initial values ​​to check for abnormalities.

[1090] Step 2:

[1091] The device periodically collects data from each sensor, such as a motion sensor that detects the presence or absence of movement, a temperature sensor that measures the indoor temperature, a smoke sensor that detects smoke, a gas sensor that detects gas leaks, and sensors that monitor the opening and closing of doors and windows, and obtains data in real time.

[1092] Step 3:

[1093] The device packages the collected sensor data at regular intervals and converts it into a format that the server can understand, such as JSON. The data is encrypted during this process.

[1094] Step 4:

[1095] The device then sends the packaged and encrypted data to the server, where the communication is performed using a secure protocol (e.g., HTTPS) to ensure the data is secure.

[1096] Step 5:

[1097] The server receives the data sent from the device and decrypts it, converting it into a format suitable for analysis.

[1098] Step 6:

[1099] The server preprocesses the received data, checking for missing values ​​and outliers, and imputes data as needed.

[1100] Step 7:

[1101] The server then runs AI analysis on the pre-processed data, using AI models (such as Isolation Forest) to detect anomalies and determine whether they match a specific pattern.

[1102] Step 8:

[1103] If an anomaly is detected, the server determines the nature of the anomaly and generates the necessary notification, which includes the type of anomaly, its location, and recommended actions to take.

[1104] Step 9:

[1105] The server then sends notifications to care providers and family members via email, SMS, or voice calls.

[1106] Step 10:

[1107] The server stores the analysis results and sensor data in a database, which can then be used for later analysis and report generation.

[1108] Step 11:

[1109] The server periodically generates and provides users with reports that include anomaly detection history, behavioral pattern analysis results, and daily health trends.

[1110] In this way, the system of the present invention achieves a series of processing flows from collecting sensor data to detecting anomalies, notifying, storing data, and generating reports.

[1111] Example 1

[1112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1113] Previously, systems designed to provide a safe and secure residential environment for the elderly and people with disabilities lacked the technology to detect abnormalities, making it difficult to detect them in real time. Furthermore, security issues existed regarding the safe transmission and storage of collected sensor data. This made it difficult to respond quickly and reliably to emergencies, and user safety was not adequately ensured.

[1114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1115] In this invention, the server includes means for receiving data from the terminal at regular intervals and encrypting the data before transmission, means for decrypting the received data, and means for analyzing the decrypted data using AI technology to detect anomalies, thereby enabling real-time anomaly detection and safe transmission and analysis of collected sensor data.

[1116] A "sensor" is a device that monitors environmental conditions and collects data.

[1117] A "motion sensor" is a sensor that detects the movement of an object and collects that information.

[1118] A "temperature sensor" is a sensor that measures the ambient temperature and collects that data.

[1119] A "smoke sensor" is a sensor that detects the presence of smoke and collects information about it.

[1120] A "gas sensor" is a sensor that detects the presence of harmful gases and collects information about them.

[1121] A "door sensor" is a sensor that detects whether a door is open or closed and collects that data.

[1122] A "window sensor" is a sensor that detects whether a window is open or closed and collects that data.

[1123] A "terminal" is a device that collects data from multiple sensors and transmits it to a server.

[1124] "Encryption" is the process of transforming data using special algorithms in order to transmit it securely.

[1125] A "server" is a computer system that receives and analyzes data sent from a terminal.

[1126] "Decryption" is the process of restoring encrypted data to its original state.

[1127] "AI technology" is a data analysis method that utilizes artificial intelligence, and is used to perform pattern recognition and anomaly detection.

[1128] A "database" is a system for storing analytical results and other information.

[1129] "Notification" is a means of sending information to care providers and family members when an abnormality is detected.

[1130] System configuration

[1131] This invention is a system for providing a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities.

[1132] Key Components

[1133] 1. Sensor group (terminal)

[1134] The device includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. These sensors monitor the environment in real time and collect data. For example, when a motion sensor detects movement, the information is collected immediately.

[1135] 2. Data transmission device (terminal)

[1136] The device sends the collected sensor data to a server at regular intervals. This process encrypts the data using an encryption algorithm such as AES (Advanced Encryption Standard). Specifically, security is ensured by sending encrypted data.

[1137] 3. Server

[1138] The server decrypts the received data and analyzes it using AI technology, such as Python machine learning libraries (e.g., TensorFlow). The server compares the decrypted data with past data to detect abnormal patterns and behavioral patterns.

[1139] 4. Notification System (Server)

[1140] If the server detects an abnormality, it will notify the care provider or family in real time via email, SMS, voice call, etc. The notification will include a detailed description of the type of abnormality and countermeasures.

[1141] Specific examples

[1142] For example, the system may be implemented in a home as follows: A motion sensor detects unusual activity in the hallway. Because this activity is unusual, the sensor collects data and the device sends it to a server. The server analyzes the received data and detects abnormal patterns. Based on the analysis, an abnormality is confirmed and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification includes the specific nature of the abnormality, the location information, and specific instructions for how to respond.

[1143] Example prompts for generative AI models

[1144] "Please describe a sensor system that provides an environment where elderly people can live safely. The system collects data from motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors, analyzes it on a server, and detects abnormalities. For example, please include how to handle abnormal activity in the hallway."

[1145] This invention will realize a home environment where the elderly and people with disabilities can live safely. The system enables early detection of abnormalities and rapid response, reducing the risk of emergencies.

[1146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1147] Step 1:

[1148] Sensor data collection

[1149] The device collects data from motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors.

[1150] As input, real-time data from each sensor is provided, and the device temporarily stores this data in its memory.

[1151] As a specific example of operation, when a motion sensor detects movement in a hallway, the detected data is stored in memory.

[1152] As an output, all collected sensor data is stored in memory.

[1153] Step 2:

[1154] Data encryption and transmission

[1155] The terminal encrypts the collected sensor data using the AES encryption algorithm and sends it to the server.

[1156] The input is the sensor data collected in step 1.

[1157] An encryption process is performed to generate encrypted data, which is then sent from the terminal to the server.

[1158] As a specific example of operation, data from a motion sensor in a hallway is encrypted and sent to a server.

[1159] As output, the encrypted sensor data is sent to the server.

[1160] Step 3:

[1161] Receiving and Decrypting Data

[1162] The server receives the data sent from the terminal and decrypts the data through a decryption process.

[1163] As input, encrypted sensor data is provided.

[1164] A decoding process is performed to reconstruct the original sensor data.

[1165] As a specific example of operation, the server receives encrypted motion sensor data and decrypts it to obtain the original data.

[1166] As an output, the decoded sensor data is obtained.

[1167] Step 4:

[1168] Data analysis

[1169] The server analyzes the decoded sensor data using AI technology.

[1170] As input, the decoded sensor data is provided.

[1171] Data analysis involves comparing data with past data and using machine learning libraries (e.g., TensorFlow) to detect behavioral and anomalous patterns.

[1172] As a specific example of operation, the server analyzes data from a motion sensor and detects movements that differ from normal behavior patterns.

[1173] The output is the analysis result, which includes a judgment of normality or abnormality.

[1174] Step 5:

[1175] Anomaly detection

[1176] The server detects anomalies based on the results of data analysis.

[1177] As input, the analysis results obtained in step 4 are provided.

[1178] If an anomaly is detected, an anomaly flag is set in the analysis results.

[1179] As a specific example of operation, the server detects abnormal movement in a hallway and adds an abnormality flag.

[1180] The output is an anomaly detection result, which is an analysis result with an anomaly flag added.

[1181] Step 6:

[1182] Send notifications

[1183] If an abnormality is detected, the server sends a notification to the care provider or family.

[1184] As input, the analysis results in which an anomaly was detected are provided.

[1185] A notification will be generated and sent via email, SMS, voice call, etc. The notification will include the specific anomaly and its location.

[1186] As a specific example of operation, the server detects an abnormality and sends a notification of the abnormality to the care provider and family via email and SMS.

[1187] As an output, a notification is sent: a notification message containing the anomaly description and location information is generated and sent to the specified contacts.

[1188] (Application example 1)

[1189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1190] In conventional residential environments, monitoring systems have been inadequate to ensure the safety of elderly and disabled people. Furthermore, there has been no system in brick-and-mortar stores that can detect and notify abnormalities in real time while ensuring the safety of elderly and disabled customers. The purpose of this invention is to solve these problems and provide an environment where elderly and disabled people can live and visit stores with peace of mind.

[1191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1192] In this invention, the server includes means for collecting data from a plurality of sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for transmitting a notification when the abnormality is detected, means for detecting environmental abnormalities with sensors installed in the store, and means for transmitting a notification to a smartphone when the environmental abnormality is detected. This not only enables elderly people and people with disabilities to live in peace of mind, but also makes it possible to ensure the safety of elderly and disabled customers in physical stores in real time.

[1193] A "sensor" is a device that acquires environmental information in real time and collects it as data.

[1194] The "data transmission means" is a means for transmitting data collected by the sensor to the server.

[1195] A "server" is a computing device that analyzes collected data and detects anomalies.

[1196] The "anomaly detection means" is a means for detecting unusual patterns or anomalies from the analyzed data.

[1197] The "notification sending means" is a means for sending a notification to relevant parties when an abnormality is detected.

[1198] A "smartphone" is a portable multi-function information terminal that has communication capabilities and can run applications.

[1199] "Inside the store" refers to the interior space of a store that sells goods.

[1200] "Environmental abnormalities" refers to situations that deviate from normal conditions, such as high temperatures, smoke, gas leaks, etc.

[1201] A "motion sensor" is a sensor for detecting movement.

[1202] This system provides a safe and secure environment for the elderly and people with disabilities, and has monitoring and notification functions using sensors, servers, and smartphones. Furthermore, this system can also be applied to similar safety systems in brick-and-mortar stores.

[1203] System configuration

[1204] Sensor group

[1205] The system includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors and window sensors, which are installed at different locations in brick-and-mortar stores and homes to monitor environmental information in real time.

[1206] Data transmission method

[1207] The data collected from the sensors is encrypted before being sent to the server, ensuring data security.

[1208] server

[1209] The server decodes the received sensor data and analyzes it using AI technology. If an anomaly is detected as a result of the analysis, a notification is triggered. The server also has the function of storing the anomaly detection results in a database.

[1210] Notification sending method

[1211] If an abnormality is detected, the server will immediately send email, SMS, or in-app notifications to relevant people (e.g., care providers and family members) via smartphone applications, ensuring the safety of seniors and people with disabilities in real time.

[1212] Specific examples

[1213] Consider a specific example of implementing the present invention in a brick-and-mortar store via a smartphone application called "Safety Navi." Sensors installed at various points in the store collect environmental information in real time, and if an abnormality is detected, the data is sent to a server.

[1214] If the server analyzes the data and detects an abnormality, the abnormality information is sent to the customer's smartphone via the app. The notification is displayed in the following format:

[1215] Prompt Sentence Examples

[1216] "A high temperature alarm has been detected near the refrigerated display case. We are instructing customers in the area to move away immediately."

[1217] This will enable elderly and disabled customers to safely visit and enjoy shopping in physical stores, and if a situation requires attention, immediate notification will enable a prompt response.

[1218] Hardware and Software Used

[1219] Sensors: Motion sensor, temperature sensor, smoke sensor, gas sensor, door sensor, window sensor

[1220] Server: Used for data analysis and anomaly detection

[1221] Smartphone: Used to receive notifications

[1222] Software: Data analysis and notification system implementation using Python, specifically using libraries such as requests, smtplib, and json.

[1223] This will make it possible to provide residential and retail environments where elderly people and people with disabilities can live safely.

[1224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1225] Step 1:

[1226] Sensors collect real-time environmental information (motion, temperature, smoke, gas, door and window data) within physical stores and homes.

[1227] Input: Physical variables of the environment (motion, temperature, smoke, gases, open and closed doors and windows).

[1228] Data processing: Sensors convert physical variables into electrical signals, which are then generated as digital data.

[1229] Output: Digital data (environmental information).

[1230] Step 2:

[1231] The device sends the collected sensor data to the server at regular intervals, and the data is encrypted to maintain security.

[1232] Input: Sensor data (digital data).

[1233] Data Computation: The sensor data is encrypted and sent to the server using a secure channel.

[1234] Output: Encrypted sensor data.

[1235] Step 3:

[1236] The server decrypts the received encrypted data and analyzes the environmental data.

[1237] Input: Encrypted sensor data.

[1238] Data computation: Decrypting the data and using AI techniques to analyze it, for example, using generative AI models to detect abnormal patterns and compare them with normal behavioral patterns.

[1239] Output: Analysis results (anomaly detection results).

[1240] Step 4:

[1241] The server generates a notification if it detects an anomaly based on the analysis results.

[1242] Input: Analysis results.

[1243] Data calculation: Based on the analysis results, a notification message is generated if an abnormality is detected.

[1244] Output: Informational message.

[1245] Step 5:

[1246] The server sends the generated notification messages to the smartphones of care providers, family members, store managers, and customers.

[1247] Input: Notification message.

[1248] Data calculation: Send notification messages as email, SMS, or in-app notifications.

[1249] Output: The notification sent.

[1250] Step 6:

[1251] The user receives a notification message via their smartphone and takes necessary measures.

[1252] Input: Notification message.

[1253] Specific actions: The user checks the contents of the message and takes action according to the specific instructions, such as avoiding a specific location or contacting an administrator.

[1254] Output: User behavior (safety action).

[1255] This will allow the entire system to be integrated and provide a safe environment for seniors, people with disabilities, and brick-and-mortar store visitors.

[1256] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1257] This invention combines an emotion engine that recognizes the user's emotions with a system that provides a home environment where the elderly and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and detects abnormalities. It also recognizes the user's emotional state and takes necessary measures.

[1258] System configuration

[1259] The system mainly consists of the following components:

[1260] 1. Sensors

[1261] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[1262] 2. Data transmission device

[1263] The device sends the collected sensor data to a server at regular intervals, and this process is performed by encrypting the data for security reasons.

[1264] 3. Server

[1265] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[1266] 4. Emotion Engine

[1267] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine uses image processing and voice analysis technologies to determine emotions from the user's facial expressions and vocalizations. This allows the server to monitor the user's mental state and notify care providers or family members as necessary.

[1268] 5. Notification System

[1269] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[1270] System Operation

[1271] The device first collects real-time data from multiple sensors placed in the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and the opening and closing status of doors and windows are continuously monitored.

[1272] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[1273] The emotion engine analyzes the user's emotions from collected video and audio data. For example, it can determine whether the user is sad or angry from their facial expression, or recognize stress levels from their tone of voice.

[1274] If an abnormality or emotional change is detected, the server will immediately notify the care provider or family member in real time, enabling a rapid response. For example, if an elderly person collapses, a fire breaks out, or the user is experiencing extreme stress, a notification with detailed information and a prompt to take action will be sent.

[1275] Specific examples

[1276] For example, the system may be implemented in a home as follows: A motion sensor detects abnormal movement in the hallway. Furthermore, a facial recognition camera analyzes the user's facial expressions and detects a sad expression. This series of data is collected by the device, encrypted, and sent to a server. The server analyzes the data and detects abnormal movement and emotional changes. As a result, an abnormality is confirmed, and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification contains the specific details of the abnormality, the user's emotional state, and specific instructions for how to respond.

[1277] In this way, a system will be realized that provides an environment where elderly people and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, this system will also monitor the user's emotional state and contribute to maintaining mental health.

[1278] The processing flow will be explained below.

[1279] Step 1:

[1280] The device initializes multiple sensors (motion sensor, temperature sensor, smoke sensor, gas sensor, door sensor, window sensor) and checks their operation. Each sensor has a self-diagnostic function to check whether it is operating normally.

[1281] Step 2:

[1282] The device periodically collects data from each sensor. For example, a motion sensor acquires information whenever it detects movement. A temperature sensor measures the room temperature at regular intervals and updates the data. Smoke and gas sensors collect data immediately when they detect an abnormality.

[1283] Step 3:

[1284] The device converts the collected sensor data into JSON-formatted packets and encrypts them, preventing unauthorized access to the data during transmission.

[1285] Step 4:

[1286] The device sends encrypted sensor data to a server using a secure protocol such as HTTPS to protect the authenticity and privacy of the data.

[1287] Step 5:

[1288] The server decrypts the received data and converts it into JSON format data, which is then pre-processed for analysis.

[1289] Step 6:

[1290] The server performs anomaly detection on the received data, using an AI model (e.g., Isolation Forest) to determine whether the collected data is within normal ranges or if an abnormal pattern has been detected.

[1291] Step 7:

[1292] The server records the results of the anomaly detection in a database, and this data is saved as history for future reference.

[1293] Step 8:

[1294] The server uses image processing technology to analyze the user's facial expressions and recognize the user's emotions (e.g., joy, anger, sadness, etc.) based on the video data sent from the camera.

[1295] Step 9:

[1296] The server uses voice analysis technology to recognize emotions from the user's voice. Based on the audio data collected from the microphone, the server analyzes the tone and pitch of the user's voice to determine their emotional state.

[1297] Step 10:

[1298] The server combines the user's emotional data analyzed by the emotion engine with sensor data for a comprehensive analysis, which allows it to determine the user's overall health and safety.

[1299] Step 11:

[1300] The server notifies care providers and family members if an abnormality or emotional change is detected. The notification includes details of the detected abnormality, the user's emotional state, and recommended actions. Notifications are sent via email, SMS, or emergency call.

[1301] Step 12:

[1302] The server stores the analysis results and notification details in a database, which will be used for future analysis and report generation.

[1303] In this way, the system of the present invention specifically implements a series of processing flows including sensor data collection, anomaly detection, emotion recognition, notification, and data storage, thereby providing comprehensive support for the safety and mental health of users.

[1304] Example 2

[1305] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1306] To provide a residential environment where elderly people and people with disabilities can live safely, it is important to detect indoor environmental changes and abnormalities early and respond quickly. However, conventional systems only collect data and detect abnormalities using sensors, and are unable to grasp the user's emotional state, making it difficult to maintain mental health and provide appropriate care. The present invention aims to solve these problems and provide a residential environment where elderly people and people with disabilities can live more safely.

[1307] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1308] In this invention, the server includes means for collecting data from a plurality of sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for the server to analyze the user's emotions, and means for transmitting a notification when the server detects the abnormality or change in emotion. This enables early detection of abnormalities and rapid response, and maintenance of mental health by understanding the user's emotional state.

[1309] "Multiple sensors" refers to different types of sensors, such as motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.

[1310] "Means for collecting data" refers to the function of using multiple sensors to monitor the indoor environmental conditions and obtain data in real time.

[1311] "Means for transmitting data to a server" refers to the function for encrypting collected sensor data and transmitting it to a server via the Internet.

[1312] "Means for the server to analyze data and detect abnormalities" refers to the function in which the server uses AI technology to analyze the sensor data it receives and compare it with normal patterns to detect abnormalities early.

[1313] "Means for the server to analyze the user's emotions" refers to the function of determining the user's emotional state based on the video and audio data collected by the server using image processing technology and audio analysis technology.

[1314] "Means for sending notifications" refers to the function of notifying care providers and family members in real time via email, SMS, voice call, etc. when an abnormality or change in the user's emotions is detected.

[1315] This invention is a system for providing a residential environment where elderly people and people with disabilities can live safely. This system collects data from multiple sensors, sends it to a server for analysis, and can detect abnormalities and recognize the user's emotional state, taking necessary measures.

[1316] System configuration

[1317] The system consists of the following components:

[1318] 1. Sensors

[1319] These include motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. installed on the device. These sensors monitor the environment in real time and collect data.

[1320] 2. Data transmission device

[1321] The device sends the collected sensor data to a server at regular intervals. This process encrypts the data for security reasons. An encryption algorithm such as AES (Advanced Encryption Standard) is used.

[1322] 3. Server

[1323] The server analyzes the received sensor data. AI technology is used to detect abnormal patterns and analyze behavioral patterns. AI models such as TensorFlow and PyTorch are used. The analysis results are automatically stored in a database, and notifications are sent promptly in the event of an emergency.

[1324] 4. Emotion Engine

[1325] The server is equipped with an emotion engine that recognizes the user's emotions. The emotion engine uses image processing technology (OpenCV, Dlib, etc.) and voice analysis technology (Google's voice recognition API, etc.) to determine emotions from the user's facial expressions and vocalizations.

[1326] 5. Notification System

[1327] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[1328] System Operation

[1329] The device first collects real-time data from multiple sensors placed in the room. For example, if a motion sensor detects movement, that information is collected immediately. Similarly, temperature, smoke, gas, and the opening and closing status of doors and windows are continuously monitored.

[1330] The collected data is encrypted and sent to a server after ensuring security. The server decrypts the received data and analyzes it for any abnormalities. During this process, AI technology is applied to compare the data with past data to detect abnormalities early. For example, abnormal behavior compared to normal behavior patterns, sudden temperature changes, or gas leaks are detected, which are deemed to be abnormalities.

[1331] The emotion engine analyzes the user's emotions from collected video and audio data. For example, it can determine whether the user is sad or angry from their facial expression, or recognize stress levels from their tone of voice.

[1332] If an abnormality or emotional change is detected, the server will immediately notify the care provider or family member in real time, enabling a rapid response. For example, if an elderly person collapses, a fire breaks out, or the user is experiencing extreme stress, a notification with detailed information and a prompt to take action will be sent.

[1333] Specific examples

[1334] For example, the system may be implemented in a home as follows: A motion sensor detects abnormal movement in the hallway. Furthermore, a facial recognition camera analyzes the user's facial expressions and detects a sad expression. This series of data is collected by the device, encrypted, and sent to a server. The server analyzes the data and detects abnormal movement and emotional changes. As a result, an abnormality is confirmed, and a notification of the abnormality is immediately sent to the care provider and family via email and SMS. The notification contains the specific details of the abnormality, the user's emotional state, and specific instructions for how to respond.

[1335] In this way, a system will be realized that provides an environment where elderly people and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, this system will also monitor the user's emotional state and contribute to maintaining mental health.

[1336] Prompt Sentence Examples

[1337] Below are some example prompts for input to a generative AI model:

[1338] Follow this prompt to explain in natural language the process of a system that provides a safe and secure living environment for the elderly and people with disabilities. This system uses multiple sensors to collect data and analyzes it on a server. It also uses an emotion engine to recognize the user's emotions and take necessary measures. Please provide specific steps and examples. Use the subject of either the server, device, or user, and use plain sentence endings.

[1339] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1340] Step 1:

[1341] The device collects data in real time from multiple sensors (motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.). The input data includes the motion, temperature, smoke, gas, and whether a door or window is open or closed, detected by the sensors. For example, when a motion sensor detects motion, that information is collected immediately and becomes input data. Specific operations include the process by which the device acquires data from the sensors. The output is the collected raw sensor data.

[1342] Step 2:

[1343] The terminal encrypts the collected sensor data. Here, an encryption algorithm such as AES (Advanced Encryption Standard) is used. The input is the raw sensor data collected in step 1. The specific operation includes the process of protecting the data using an encryption algorithm. The output is the encrypted sensor data.

[1344] Step 3:

[1345] The terminal sends the encrypted sensor data to the server via the Internet. The input is the sensor data encrypted in step 2. The specific operation includes a process of sending the data to the server using a communication protocol. The output is the encrypted data sent to the server.

[1346] Step 4:

[1347] The server receives and decrypts the encrypted data sent over the Internet. The input is the encrypted data sent in step 3. The specific operations include the process of decrypting the data using a decryption algorithm. The output is the decrypted sensor data.

[1348] Step 5:

[1349] The server analyzes the decoded sensor data and detects abnormal patterns. This is done using AI technology. The input is the sensor data decoded in step 4. Specific operations include analyzing the data using AI models such as TensorFlow and PyTorch to detect abnormalities. The output is the presence or absence of anomalies and the analysis results.

[1350] Step 6:

[1351] The server analyzes the user's emotions using the decoded video and audio data. The emotion engine uses image processing technologies (such as OpenCV and Dlib) and audio analysis technologies (such as Google's speech recognition API). The input is video and audio data. Specific operations include identifying the user's emotions from facial expressions and tone of voice. The output is the identified emotional state of the user.

[1352] Step 7:

[1353] The server notifies care providers and families if it detects anomalies or emotional changes. The inputs are the anomaly information detected in step 5 and the emotional state identified in step 6. Specific operations include sending information in real time via the notification system using email, SMS, or voice calls. The output is a notification sent to the care provider or family.

[1354] (Application example 2)

[1355] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1356] In order to provide an environment where the elderly and people with disabilities can live safely, it is important not only to detect abnormalities in the environment but also to monitor the user's emotional state and provide prompt care as needed. However, current systems are specialized in detecting abnormalities, and therefore have the problem of not being able to respond appropriately based on the user's emotional state.

[1357] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from multiple sensors, means for transmitting the data to the server, means for the server to analyze the data and detect abnormalities, means for transmitting a notification when the abnormality is detected, means for the server to analyze image data and voice data of the user to recognize emotions, and means for transmitting a notification according to the user's emotional state. This makes it possible to monitor the user's emotional state in addition to detecting abnormalities in the environment, and to respond quickly and appropriately.

[1358] The "means for collecting data from multiple sensors" is a device that acquires environmental data using various types of sensors such as motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, and window sensors.

[1359] The "means for transmitting the data to the server" refers to a communication device and process for securely transmitting the collected sensor data to the server via the Internet or a dedicated line.

[1360] The "means by which the server analyzes the data and detects anomalies" refers to a system that uses artificial intelligence (AI) or machine learning algorithms to analyze data received on the server and detect unusual patterns or changes in the environment.

[1361] The "means for sending a notification when the abnormality is detected" refers to a system that, when the server confirms an abnormality, notifies pre-defined contacts (e.g., care providers or family members) of the details of the abnormality via email, SMS, voice call, etc.

[1362] "Means for recognizing emotions by analyzing user image and audio data" refers to artificial intelligence (AI) technology that analyzes video and audio data acquired through cameras and microphones, and determines the user's emotional state from their facial expressions and tone of voice.

[1363] The "means for sending notifications according to the user's emotional state" is a system that notifies care providers and family members in real time as needed based on the recognized emotions.

[1364] This invention is a system that provides a residential environment where elderly people and people with disabilities can live safely. Specifically, it collects data from multiple sensors, sends it to a server for analysis, and can detect abnormalities and recognize the user's emotional state. This system consists of the following main components:

[1365] 1. Sensors

[1366] The sensor group includes motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc. These sensors are installed on the terminal to monitor the indoor environment in real time and collect data.

[1367] 2. Data transmission device

[1368] The device sends the collected sensor data to the server at regular intervals. The data is encrypted for security reasons.

[1369] 3. Server

[1370] The server analyzes the received sensor data. This analysis utilizes AI technology to detect abnormal patterns and analyze behavioral patterns. The analysis results are automatically saved in a database and have the function of sending prompt notifications in the event of an emergency.

[1371] 4. Emotion Engine

[1372] The server is equipped with an emotion engine that recognizes the user's emotions. This emotion engine uses image processing and voice analysis technologies to determine emotions from the user's facial expressions and voice, monitors the user's mental state, and notifies care providers or family members as necessary.

[1373] 5. Notification System

[1374] When the server detects an abnormality or the user's emotional state, it notifies the care provider or family via email, SMS, voice call, etc. The notification content includes the type of abnormality or emotional state and countermeasures.

[1375] As a concrete example of how this might work, the system might operate as follows:

[1376] In a home, when a motion sensor detects abnormal activity in the hallway, the data is collected by the device, encrypted, and sent to a server. A camera then captures the user's facial expressions, which an emotion engine analyzes to detect sadness. This data is then analyzed by the server, which identifies abnormalities and emotional changes. The server then sends an email and SMS notification to the care provider and family members. The notification describes the abnormality, the user's emotional state, and includes specific instructions for how to respond.

[1377] In this way, a system will be realized that provides an environment where the elderly and people with disabilities can live with peace of mind. In addition to early detection of abnormalities and prompt response, the system will also monitor the user's emotional state and contribute to maintaining mental health.

[1378] Example prompts for generative AI models

[1379] Read the code below and write a Python program that collects data from the camera and sensors, analyzes the user's emotions, and sends a notification if an abnormality occurs. Use OpenCV, requests, cryptography, and smtplib for the program.

[1380] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1381] Step 1:

[1382] The device collects real-time environmental data from multiple sensors (motion sensors, temperature sensors, smoke sensors, gas sensors, door sensors, window sensors, etc.). This input data includes measurements from each sensor (e.g., temperature, motion detection, gas concentration, etc.). The device temporarily stores the collected data and prepares it for the next step.

[1383] Step 2:

[1384] The device sends the collected sensor data to the server at regular intervals. During this process, the device encrypts the data using a secure communication protocol (e.g., TLS). The data includes measurements from each sensor, and the server receives and analyzes it.

[1385] Step 3:

[1386] The server analyzes the received sensor data using AI technology and machine learning algorithms. This analysis process identifies patterns in the data and compares them with normal behavior and environmental conditions to detect anomalies. The analysis results include whether an anomaly occurred and the type of anomaly detected.

[1387] Step 4:

[1388] The server analyzes the user's image and audio data collected from the camera and microphone to recognize emotions. The emotion engine uses image processing and audio analysis technologies to determine the user's emotional state (e.g., sadness, anger, happiness, etc.) from their facial expressions and tone of voice. The input for this process is video and audio data, and the output is the recognized emotional state.

[1389] Step 5:

[1390] The server continuously monitors the user's emotional state and detects abnormalities based on the analysis results. If an abnormality or a specific emotional state is detected, the server sends a notification to pre-defined care providers or family members. The notification includes the details of the abnormality, the user's emotional state, and countermeasures. The notification can be sent via email, SMS, voice call, or other methods.

[1391] Step 6:

[1392] The server stores the analysis results and notification history in a database. The stored data includes measurements from each sensor, the recognized emotional state, and the content of the sent notification. This allows the system to track the user's long-term behavioral patterns and emotional state and use them for future analysis and countermeasures.

[1393] Step 7:

[1394] Users can change system settings and update notification destinations as needed, allowing the system to respond flexibly to individual user needs. User input to change settings is sent to the server and reflected in the overall system operation.

[1395] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1396] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1397] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1398] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1399] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1400] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1401] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1402] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1403] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1404] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1405] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1406] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1407] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1408] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1409] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1410] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1411] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1412] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1413] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1414] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1415] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1416] The following is further disclosed regarding the above embodiment.

[1417] (Claim 1)

[1418] a means for collecting data from a plurality of sensors;

[1419] means for transmitting said data to a server;

[1420] means for the server to analyze the data and detect anomalies;

[1421] means for sending a notification when the abnormality is detected;

[1422] A system including:

[1423] (Claim 2)

[1424] The system of claim 1 , wherein the sensors include a motion sensor, a temperature sensor, a smoke sensor, a gas sensor, a door sensor, and a window sensor.

[1425] (Claim 3)

[1426] The system of claim 1 , further comprising means for the server to receive the data in an encrypted form and store the analysis results.

[1427] "Example 1"

[1428] (Claim 1)

[1429] a means for collecting data from a plurality of sensors;

[1430] means for transmitting said data to a server at regular intervals and encrypting said data before transmission;

[1431] A means for decoding the data received by the server, analyzing the data using AI technology, and detecting anomalies;

[1432] means for sending a notification when the abnormality is detected;

[1433] A system including:

[1434] (Claim 2)

[1435] The system of claim 1 , wherein the sensors include a motion sensor, a temperature sensor, a smoke sensor, a gas sensor, a door sensor, and a window sensor.

[1436] (Claim 3)

[1437] 10. The system of claim 1, further comprising means for the server to use AI techniques to compare the decrypted data with past data when analyzing the decrypted data, and to store the analysis results in a database.

[1438] "Application Example 1"

[1439] (Claim 1)

[1440] a means for collecting data from a plurality of sensors;

[1441] means for transmitting said data to a server;

[1442] means for the server to analyze the data and detect anomalies;

[1443] means for sending a notification when the abnormality is detected;

[1444] A means of detecting environmental abnormalities using sensors installed in the store,

[1445] means for sending a notification to a smartphone when an abnormality in the environment is detected;

[1446] A system including:

[1447] (Claim 2)

[1448] The system of claim 1 , wherein the sensors include a motion sensor, a temperature sensor, a smoke sensor, a gas sensor, a door sensor, and a window sensor.

[1449] (Claim 3)

[1450] The system of claim 1 , further comprising means for the server to receive the data in an encrypted form and store the analysis results.

[1451] "Example 2: Combining Emotion Engines"

[1452] (Claim 1)

[1453] a means for collecting data from a plurality of sensors;

[1454] means for transmitting said data to a server;

[1455] means for the server to analyze the data and detect anomalies;

[1456] A means for the server to analyze user emotions;

[1457] means for sending a notification when said abnormality or change in emotion is detected;

[1458] A system including:

[1459] (Claim 2)

[1460] The system of claim 1 , wherein the sensors include a motion sensor, a temperature sensor, a smoke sensor, a gas sensor, a door sensor, and a window sensor.

[1461] (Claim 3)

[1462] The system of claim 1 , further comprising means for the server to receive the data in an encrypted form and store the analysis results.

[1463] "Application example 2 when combining emotion engines"

[1464] (Claim 1)

[1465] a means for collecting data from a plurality of sensors;

[1466] means for transmitting said data to a server;

[1467] means for the server to analyze the data and detect anomalies;

[1468] means for sending a notification when the abnormality is detected;

[1469] means for the server to analyze image data and voice data of a user and recognize emotions;

[1470] means for sending notifications in response to the emotional state of the user;

[1471] A system including:

[1472] (Claim 2)

[1473] The system of claim 1 , wherein the sensors include a motion sensor, a temperature sensor, a smoke sensor, a gas sensor, a door sensor, and a window sensor.

[1474] (Claim 3)

[1475] The system of claim 1 , further comprising means for the server to receive the data in an encrypted form and store the analysis results. [Explanation of symbols]

[1476] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting data from a plurality of sensors; means for transmitting said data to a server; means for the server to analyze the data and detect anomalies; means for sending a notification when the abnormality is detected; A system including:

2. The system of claim 1 , wherein the sensors include a motion sensor, a temperature sensor, a smoke sensor, a gas sensor, a door sensor, and a window sensor.

3. The system of claim 1 , further comprising means for the server to receive the data in an encrypted form and store the analysis results.

Images