System

Abstract

A system is provided.SOLUTION: A system including means for collecting sensor data, means for learning a behavior pattern of a user using artificial intelligence based on the collected sensor data, means for generating an action suitable for the user based on the learned behavior pattern, means for controlling a device for executing the generated action, means for notifying a caregiver when a specific event occurs, and means for recording all actions and events as a log.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Elderly people with dementia and their caregivers face many challenges in their daily lives, from simple household tasks that can be easily forgotten to decisions that pose safety risks. Furthermore, constantly monitoring the patient's safety places a significant burden on family members and caregivers. The present invention aims to help dementia patients live independently and safely at home, while reducing the burden on caregivers. [Means for solving the problem]

[0005] The present invention solves this problem by providing a system that includes a means for collecting sensor data, a means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, a means for generating actions appropriate for the user based on the learned behavioral patterns, a means for controlling a device to execute the generated actions, a means for notifying a caregiver when a specific event occurs, and a means for recording all actions and events as a log. The collected sensor data includes temperature data, lighting data, and door opening / closing status data. Based on the learned behavioral patterns, the system automatically turns on the lights when the user wakes up in the middle of the night, maintains an appropriate temperature, and issues an alert if a door is opened or closed incorrectly, thereby providing a safe and comfortable living environment for dementia patients.

[0006] "Sensor Data" means information collected by sensors to monitor a user's environment and behavior.

[0007] "Artificial intelligence" is technology built into computer programs or systems that mimics human intelligence to analyze data and make decisions.

[0008] "Behavioral patterns" refer to a series of actions and habits in the user's daily life, and are used to predict certain behavioral tendencies of the user.

[0009] "Action" refers to a specific operation or behavior generated by the system to ensure the user's comfort and safety.

[0010] "Devices" are appliances and devices installed in a user's living environment, including lights, thermostats, and door locks.

[0011] An "event" is an occurrence or occurrence that the system recognizes based on certain conditions, and whose occurrence triggers an action.

[0012] "Log" refers to a record of all actions taken or events detected by a system, which can later be used for analysis or auditing.

[0013] "Incorrect door opening and closing" refers to an action in which a user accidentally opens or closes a door without intending to do so, which is considered to pose a high safety risk. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The system of the present invention is a smart home solution for dementia patients that automatically adjusts various aspects of daily life, such as lighting, temperature, and security systems, to provide a comfortable and safe living environment.

[0036] Program Overview

[0037] The system consists of the following main modules:

[0038] 1. Sensor data collection module

[0039] 2. AI Learning Module

[0040] 3. Action Generation Module

[0041] 4. Notification Module

[0042] 5. Log Management Module

[0043] Explaining program processing in natural language

[0044] Sensor Data Collection Module

[0045] The server collects real-time data from various sensors (temperature sensors, lighting sensors, door sensors, etc.) installed in the user's environment, including temperature, lighting status, and door open / close status.

[0046] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0047] AI Learning Module

[0048] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[0049] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[0050] Action Generation Module

[0051] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[0052] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[0053] Notification Module

[0054] The server can send alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[0055] Example: If a user accidentally opens an external door in the middle of the night, the server immediately sends a notification to the device to notify the caregiver.

[0056] Log Management Module

[0057] The server logs all actions and sensor data, making it available for later analysis and troubleshooting.

[0058] Example: A server records detailed data such as which door a user opens and at what time, and which lights are turned on and at what time, and stores this information for future reference.

[0059] Specific example details

[0060] 1. Nighttime scenario

[0061] Consider a situation where a user wakes up at 2am to go to the bathroom.

[0062] The server receives data from the door sensors in real time.

[0063] The server immediately switches on the lights smoothly, allowing the user to move safely.

[0064] The server also obtains temperature data from the room's temperature sensor and issues instructions to adjust the temperature to the appropriate level.

[0065] If the door remains open for a long period of time, the server determines that it is an erroneous door opening and closing and instantly sends an alert to the terminal.

[0066] The terminal notifies the caregiver of the received alert.

[0067] Finally, the server logs every step in detail and stores it for future reference.

[0068] In this way, the system reduces the burden on caregivers while ensuring the comfort and safety of the user.

[0069] The processing flow will be explained below.

[0070] Program processing flow

[0071] Sensor Data Collection Module

[0072] Step 1:

[0073] The server periodically requests data from temperature sensors, light sensors, door sensors, etc.

[0074] Step 2:

[0075] The server receives the real-time data sent from each sensor.

[0076] Step 3:

[0077] The server stores the received data in a database.

[0078] AI Learning Module

[0079] Step 1:

[0080] The server retrieves historical sensor data from the database and performs preprocessing, which includes imputing missing values ​​and normalizing the data.

[0081] Step 2:

[0082] The server runs machine learning algorithms on the prepared data to learn user behavior patterns.

[0083] Step 3:

[0084] The server updates the obtained model and accumulates behavioral patterns.

[0085] Action Generation Module

[0086] Step 1:

[0087] The server assesses the user's current state based on the sensor data collected in real time, for example, determining whether the user is asleep or awake.

[0088] Step 2:

[0089] The server generates appropriate actions (e.g., adjusting the lights or changing the temperature) based on the learned behavioral patterns.

[0090] Step 3:

[0091] The server sends instructions to the corresponding devices to execute the generated actions.

[0092] Notification Module

[0093] Step 1:

[0094] The server monitors whether a particular event (e.g., a suspicious door opening or closing) has occurred.

[0095] Step 2:

[0096] The server generates an alert when an event occurs that meets the set conditions.

[0097] Step 3:

[0098] The server transmits the generated alert to the caregiver's terminal.

[0099] Step 4:

[0100] The device notifies the caregiver of the received alert so that they can understand the situation.

[0101] Log Management Module

[0102] Step 1:

[0103] The server generates a log of all actions and events.

[0104] Step 2:

[0105] The server stores the generated logs in a database with a timestamp.

[0106] Step 3:

[0107] The server periodically analyzes logs to monitor system performance, detect abnormalities, and identify areas for improvement.

[0108] Specific example details

[0109] For nighttime scenarios:

[0110] 1. The user wakes up at 2am to go to the bathroom.

[0111] Step 1:

[0112] The server receives door opening / closing data from the door sensor at 2 a.m.

[0113] Step 2:

[0114] The server stores the data in a database and begins processing it in real time.

[0115] Step 3:

[0116] The server recognizes that the user is moving from the bedroom to the bathroom and immediately sends a command to the lighting device to turn on the lights.

[0117] Step 4:

[0118] The server simultaneously obtains temperature data from the room's temperature sensor and sends instructions to the heating or cooling device to adjust the temperature to the desired level.

[0119] Step 5:

[0120] The server determines that the door has been opened or closed incorrectly and quickly sends an alert to the terminal.

[0121] Step 6:

[0122] The device notifies the caregiver of the received alert so that they can understand the situation.

[0123] Step 7:

[0124] The server logs every step in detail and stores it for future analysis.

[0125] This system ensures the comfort and safety of the user while reducing the burden on caregivers.

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] Maintaining a comfortable and safe living environment for dementia patients places a heavy burden on caregivers. There is a need for an efficient system that can predict users' behavioral patterns and automatically generate and execute appropriate actions. The goal of this system is to reduce the burden on caregivers while ensuring the safety and comfort of users.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes means for collecting data from multiple sensors installed in the environment, means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, means for generating actions appropriate for the user based on the learned behavioral patterns, means for controlling devices to execute the generated actions, means for notifying a caregiver when a specific event occurs, and means for recording all actions and events as a log. This makes it possible to predict the user's behavior and take appropriate measures to ensure the user's safety and comfort and reduce the burden on the caregiver.

[0131] "Sensor data" is a general term for various data such as temperature, lighting, and door opening / closing status obtained from sensors installed in the environment.

[0132] "Artificial intelligence" is a technology that allows computers to imitate human intelligence, learn, reason, and assist in problem-solving.

[0133] A "behavior pattern" is a collection of data that indicates the tendencies and regularities of the user's daily behavior.

[0134] An "action" is a response or operation that the system automatically generates and executes based on the user's behavioral patterns.

[0135] A "device" is a hardware device controlled by the system, and specifically includes lighting devices, temperature control devices, door lock devices, etc.

[0136] An "event" refers to a specific action or change in state of the user, such as opening and closing a door, a sudden change in temperature, or movement during the night.

[0137] "Notifications" are alerts or messages sent to caregivers when certain events occur.

[0138] "Log" means detailed historical data of all actions and events recorded by the system for later analysis and troubleshooting.

[0139] A "caregiver" is a person whose role is to support the lives of people with dementia and provide the necessary care.

[0140] "Environment" refers to the residence where the user lives and the surrounding area, and refers to the space where the sensor is installed.

[0141] This invention relates to a smart home solution for dementia patients. The system automatically adjusts various aspects of daily life such as lighting, temperature, and security systems to provide a comfortable and safe living environment.

[0142] Program Overview

[0143] The system includes the following main modules:

[0144] 1. Sensor data collection module

[0145] 2. AI Learning Module

[0146] 3. Action Generation Module

[0147] 4. Notification Module

[0148] 5. Log Management Module

[0149] Sensor Data Collection Module

[0150] Users install temperature sensors, lighting sensors, door sensors, etc. in their living spaces. These sensors collect the user's environmental data (temperature, lighting status, door open / close status, etc.) in real time. This sensor data is sent to a server and recorded.

[0151] As a specific example, when a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0152] AI Learning Module

[0153] The server uses collected sensor data to run a learning module using artificial intelligence (AI) to learn the user's behavioral patterns and environmental preferences, making it possible to predict the user's behavior at specific times and in specific situations.

[0154] As a specific example, a server that learns that a user moves to their bedroom around 10 p.m. every night will set the room temperature to be adjusted appropriately during that time.

[0155] Action Generation Module

[0156] The server generates appropriate actions based on the behavioral patterns obtained from the AI ​​learning module, such as automatically turning on the lights when the user wakes up.

[0157] As a specific example, the server learns that the user wakes up at 7:00 every morning and sets the lights to gradually brighten accordingly.

[0158] Notification Module

[0159] The server sends alerts and notifications to the caregiver's device when certain events occur, allowing the caregiver to understand the user's real-time situation and respond quickly.

[0160] As a specific example, if a user accidentally opens an external door in the middle of the night, the server quickly detects this information and sends a notification to the caregiver's terminal.

[0161] Log Management Module

[0162] The server records detailed logs of all actions and events, which can be used for later analysis and troubleshooting.

[0163] For example, the server records detailed data such as which door the user opened at what time, and which lights were turned on at what time, and stores this information for future reference.

[0164] Specific example details

[0165] 1. Nighttime scenario

[0166] If the user wakes up at 2am to go to the bathroom

[0167] The server receives data from the door sensors in real time.

[0168] The server instantly switches on the lights smoothly to allow the user to move safely.

[0169] The server obtains temperature data from the temperature sensor in the room and issues instructions to adjust the temperature to the appropriate level.

[0170] If the door is left open for a long period of time, the server determines that it is an erroneous door opening and closing and instantly sends an alert to the caregiver's device.

[0171] The device notifies the caregiver of the received alert.

[0172] Finally, the server logs every step in detail and stores it for future reference.

[0173] Examples of prompt statements

[0174] Below are some example prompts to input to a generative AI model:

[0175] "Describe a smart home system that automatically turns on the lights, maintains the appropriate room temperature, and notifies caregivers as needed when a user opens the door to go to the bathroom at night."

[0176] In this way, the present invention aims to reduce the burden on caregivers while ensuring the comfort and safety of the user.

[0177] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0178] Specific processing flow of the program

[0179] Sensor Data Collection Module

[0180] Step 1:

[0181] Users install temperature sensors, light sensors, door sensors, etc. in their living spaces to monitor the user's behavior and environmental conditions in real time.

[0182] Input: Sensor installation area information

[0183] Operation: Sensor device installation and initial setup

[0184] Step 2:

[0185] The server collects real-time data from each sensor, including the room temperature, lighting status, and door open / close status.

[0186] Input: Real-time data from each sensor

[0187] What it does: polls and collects data

[0188] Output: Real-time sensor data

[0189] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0190] AI Learning Module

[0191] Step 3:

[0192] The server runs a learning module using artificial intelligence (AI) based on the collected sensor data. The AI ​​module learns the user's behavioral patterns and makes it possible to predict the user's behavior at specific times and in specific situations.

[0193] Input: Real-time sensor data

[0194] What it does: Analyze data and run learning algorithms

[0195] Output: Learned behavioral patterns

[0196] Example: AI learns the pattern of a user moving to the bedroom around 10 p.m. each night and adjusts the room temperature appropriately at that time.

[0197] Action Generation Module

[0198] Step 4:

[0199] The server generates optimal actions for users based on behavioral patterns obtained from the AI ​​learning module, such as automatically turning on lights or adjusting the temperature.

[0200] Input: Learned behavioral patterns

[0201] Behavior: Condition determination and action generation

[0202] Output: The generated actions

[0203] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[0204] Notification Module

[0205] Step 5:

[0206] The server sends alerts and notifications to the caregiver's device when certain events occur, allowing the caregiver to understand the user's current condition in real time and respond quickly to emergencies.

[0207] Input: A specific event (e.g., door opening and closing at midnight)

[0208] Action: Detecting an event and sending a notification

[0209] Output: Notification sent to caregiver's device

[0210] Example: If a user accidentally opens an external door late at night, the server detects this information and promptly sends a notification to the caregiver's device.

[0211] Log Management Module

[0212] Step 6:

[0213] The server records detailed logs of all actions and events, which are stored for later analysis and troubleshooting.

[0214] Input: Generated actions and detected events

[0215] Operation: Recording and storing data

[0216] Output: Detailed log data

[0217] Example: A server records detailed data such as which door a user opens and at what time, and which lights are turned on and at what time, and stores this information for future reference.

[0218] Thus, each processing step of this program aims to reduce the burden on caregivers while ensuring the comfort and safety of the user.

[0219] (Application example 1)

[0220] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0221] Current physical store management systems lack the ability to automatically adjust environmental factors such as temperature and lighting according to customer visits, resulting in an inconsistent customer experience and a heavy burden on managers to make manual adjustments. Notification functions for abnormalities are also limited, making it difficult to respond quickly. This hinders efficient store operations and improved customer satisfaction.

[0222] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0223] In this invention, the server includes means for collecting sensor data, means for learning user behavior patterns based on the collected sensor data using artificial intelligence, means for generating actions appropriate for the user based on the learned behavior patterns, means for controlling devices to execute the generated actions, means for notifying an administrator when a specific event occurs, means for recording all actions and events as a log, and means for automatically adjusting environmental factors (temperature, lighting, etc.) in the physical store according to customer behavior patterns, thereby enabling effective environmental management and rapid response to abnormalities.

[0224] "Sensor data" refers to environmental information collected by sensors, such as temperature, lighting, human presence, and door opening and closing status.

[0225] "Artificial intelligence" is the technology that enables computer systems to analyze data patterns, learn, and make predictions.

[0226] A "behavioral pattern" is a collection of data that indicates the tendency of regular behavior of a user or customer.

[0227] "Action" refers to optimal behavior or adjustments generated based on learned behavioral patterns.

[0228] "Device" is a general term for equipment or facilities that are controlled to execute the generated action.

[0229] "Administrator" refers to the person in charge of running the system or store, and who will be notified of events.

[0230] A "log" is a data record that records all actions and events and is used for later analysis and reference.

[0231] The system of the present invention is a smart environment management system that aims to improve the customer experience in physical stores and streamline store management. This system is composed of the following main modules.

[0232] 1. Sensor data collection module

[0233] The server collects data in real time from temperature sensors, lighting sensors, motion sensors, door sensors, etc. installed in the physical store, thereby obtaining environmental information within the store (temperature, lighting, human movement, door opening / closing status).

[0234] 2. AI Learning Module

[0235] The server uses artificial intelligence to learn customer behavior patterns based on collected sensor data. This module uses data analysis and pattern recognition to predict customer movements within the store at specific times.

[0236] 3. Action Generation Module

[0237] The server generates optimal actions based on learned behavioral patterns, such as automatically brightening the lights and maintaining an appropriate temperature when the store is crowded.

[0238] 4. Device Control Module

[0239] The server controls devices (such as lighting equipment, air conditioning equipment, and security devices) to execute the generated actions, thereby providing a comfortable environment for the customer.

[0240] 5. Notification Module

[0241] The server sends an alert to the administrator's device when a specific event occurs. For example, if it detects abnormal activity, it will immediately notify the administrator.

[0242] 6. Log Management Module

[0243] The server records all actions and events in detail and makes them available for later analysis and troubleshooting.

[0244] Hardware and software used

[0245] Hardware: Temperature sensors, lighting sensors, motion sensors, door sensors, lighting equipment, air conditioning equipment, security devices

[0246] Software: Python, data collection API, notification API, log storage function

[0247] Specific examples

[0248] For example, consider the process when a particular area in a store is crowded.

[0249] The server receives congestion data from the motion sensors in real time.

[0250] The server uses an AI learning module to analyze congestion patterns.

[0251] The server uses an action generation module to generate actions to brighten the lights and lower the temperature.

[0252] The server controls lighting fixtures and air conditioning equipment through device control modules.

[0253] As a result of this series of actions, customers can visit the store in a more comfortable environment.

[0254] Prompt Sentence Examples

[0255] For example, by inputting the following prompt sentence into the generative AI model, it is possible to generate a document that improves the action generation algorithm.

[0256] "Please provide a detailed description of appropriate environmental adjustment methods when stores are crowded, based on data obtained from motion sensors and temperature sensors in the physical store."

[0257] Based on this prompt, the generative AI can suggest specific actions and adjustments.

[0258] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0259] Step 1:

[0260] The server collects data in real time from temperature sensors, lighting sensors, motion sensors, door sensors, and other sensors installed in the physical store. Environmental information from each sensor (temperature, lighting, human movement, door opening / closing status) is taken in as input, and sensor data is obtained as output. This sensor data makes it possible to grasp the current situation inside the store.

[0261] Step 2:

[0262] The server uses artificial intelligence (AI) to learn customer behavior patterns based on the collected sensor data. The sensor data is passed as input to an AI learning module, which performs data analysis and pattern recognition. The output includes customer behavior patterns and trends, making it possible to predict customer behavior at specific times and in specific situations.

[0263] Step 3:

[0264] The server generates optimal actions based on the learned behavioral patterns. It uses the behavioral patterns obtained from the AI ​​learning module and real-time sensor data as input. It performs data calculations (adjusting temperature, lighting, etc.) and obtains specific actions as output. These actions include automatically adjusting the temperature and lighting in the store.

[0265] Step 4:

[0266] The server controls devices (such as lighting equipment, air conditioning equipment, and security devices) to execute the generated actions. It receives action instructions from the action generation module as input and sends specific signals to control each device as output. This operation makes the in-store environment more comfortable for customers.

[0267] Step 5:

[0268] The server sends an alert to the administrator's terminal when a specific event occurs. It monitors sensor data and system status as input, and generates notification data as output when an abnormality or specific condition is detected, which is sent to the administrator's terminal. This allows the administrator to quickly respond to the abnormality.

[0269] Step 6:

[0270] The server records all actions and events in detail and saves them as log data. It collects sensor data, action data, notification data, etc. as input and records them in a log file as output. This log data can be used later for analysis and troubleshooting.

[0271] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0272] The system of the present invention combines a smart home solution for dementia patients with an emotion engine that recognizes the user's emotions. The system automatically adjusts various aspects of daily life, such as lighting, temperature, and security systems, to provide a comfortable and safe living environment based on the user's emotional state.

[0273] Program Overview

[0274] The system consists of the following main modules:

[0275] 1. Sensor data collection module

[0276] 2. AI Learning Module

[0277] 3. Action Generation Module

[0278] 4. Emotion Recognition Module

[0279] 5. Notification Module

[0280] 6. Log Management Module

[0281] Explaining program processing in natural language

[0282] Sensor Data Collection Module

[0283] The server periodically collects data from temperature sensors, light sensors, door sensors, etc. This data is used to monitor the user's environment and for subsequent processing.

[0284] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0285] AI Learning Module

[0286] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[0287] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[0288] Action Generation Module

[0289] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[0290] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[0291] Emotion Recognition Module

[0292] The server uses an emotion engine that analyzes the user's facial expressions, tone of voice, and biometric data to recognize the user's emotions. Recognized emotions include stress, anxiety, joy, etc.

[0293] Example: If a user repeatedly engages in stressful behavior, the server recognizes the pattern through an emotion engine and generates corresponding actions.

[0294] Notification Module

[0295] The server sends alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[0296] Example: If the emotion engine recognizes that the user is in a stressful state, the server sends a notification to the terminal to notify the caregiver.

[0297] Log Management Module

[0298] The server logs all actions, sensor data, and emotion recognition results, making them available for later analysis and troubleshooting.

[0299] Example: A server logs details such as which door a user opens at what time, which lights are turned on at what time, and how the user's emotional state changes, and stores this information for future reference.

[0300] Specific example details

[0301] 1. Nighttime scenario

[0302] Consider a situation where a user wakes up at 2am to go to the bathroom.

[0303] Step 1:

[0304] The server receives door opening / closing data from the door sensor at 2 a.m.

[0305] Step 2:

[0306] The server instantly switches on the lights smoothly, allowing the user to move safely.

[0307] Step 3:

[0308] The server obtains temperature data from the temperature sensor in the room and sends instructions to adjust the temperature to the appropriate level.

[0309] Step 4:

[0310] The server uses an emotion recognition module to check whether the user's emotions change during the process of going to the toilet.

[0311] Step 5:

[0312] If the user is feeling stressed or anxious, the server notifies the caregiver's terminal of that information.

[0313] Step 6:

[0314] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[0315] Step 7:

[0316] The server logs every step in detail and stores it for later analysis.

[0317] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[0318] The processing flow will be explained below.

[0319] Program processing flow

[0320] Sensor Data Collection Module

[0321] Step 1:

[0322] The server periodically requests data from temperature sensors, light sensors, and door sensors.

[0323] Step 2:

[0324] The server receives the real-time data sent from each sensor.

[0325] Step 3:

[0326] The server stores the received data in a database.

[0327] AI Learning Module

[0328] Step 1:

[0329] The server retrieves historical sensor data from the database and performs preprocessing, which includes imputing missing values ​​and normalizing the data.

[0330] Step 2:

[0331] The server runs machine learning algorithms on the prepared data to learn user behavior patterns.

[0332] Step 3:

[0333] The server updates the obtained model and accumulates behavioral patterns.

[0334] Action Generation Module

[0335] Step 1:

[0336] The server assesses the user's current state based on the sensor data collected in real time, for example, determining whether the user is asleep or awake.

[0337] Step 2:

[0338] The server generates appropriate actions (e.g., adjusting the lights or changing the temperature) based on the learned behavioral patterns.

[0339] Step 3:

[0340] The server sends instructions to the corresponding devices to execute the generated actions.

[0341] Emotion Recognition Module

[0342] Step 1:

[0343] The server collects the user's facial expressions, tone of voice, and biometric data (e.g., heart rate and skin potential) in real time.

[0344] Step 2:

[0345] The server analyzes the collected data through an emotion engine to recognize the user's emotion.

[0346] Step 3:

[0347] The server modifies or corrects system actions and notifications based on the recognized emotions.

[0348] Notification Module

[0349] Step 1:

[0350] The server monitors whether a specific event has occurred (for example, a suspicious door opening or the user's state of stress).

[0351] Step 2:

[0352] The server generates an alert when an event that meets the condition occurs.

[0353] Step 3:

[0354] The server transmits the generated alert to the caregiver's terminal.

[0355] Step 4:

[0356] The device notifies the caregiver of the received alert so that they can understand the situation.

[0357] Log Management Module

[0358] Step 1:

[0359] The server generates logs of all actions, events, and emotion recognition results.

[0360] Step 2:

[0361] The server stores the generated logs in a database with a timestamp.

[0362] Step 3:

[0363] The server periodically analyzes logs to monitor system performance, detect abnormalities, and identify areas for improvement.

[0364] Specific example details

[0365] 1. Nighttime scenario

[0366] The user wakes up at 2am to go to the bathroom.

[0367] Step 1:

[0368] The server receives door opening / closing data from the door sensor at 2 a.m.

[0369] Step 2:

[0370] The server stores the data in a database and begins processing it in real time.

[0371] Step 3:

[0372] The server recognizes that the user is moving from the bedroom to the bathroom and immediately sends a command to the lighting device to turn on the lights.

[0373] Step 4:

[0374] At the same time, the server obtains temperature data from the room's temperature sensor and sends instructions to the heating or cooling device to adjust the temperature to the appropriate level.

[0375] Step 5:

[0376] The server collects the user's facial expressions, tone of voice, and biometric data through an emotion recognition module to recognize the user's emotions.

[0377] Step 6:

[0378] If the user feels stress or anxiety while going to the toilet, the server notifies the caregiver's terminal of that information in real time.

[0379] Step 7:

[0380] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[0381] Step 8:

[0382] The server logs every step in detail and stores it for later analysis.

[0383] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[0384] Example 2

[0385] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0386] In modern times, smart home systems for dementia patients are important for supporting their daily lives. However, conventional smart home systems have limited automation capabilities based on the user's behavioral patterns and lack the ability to respond to the user's emotional changes. Therefore, there is a need for a system that can monitor the emotional state of dementia patients in real time and adjust the environment accordingly.

[0387] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0388] In this invention, the server includes means for collecting sensor data, means for learning user behavior patterns based on the collected sensor data using artificial intelligence, and means for generating actions suitable for the user based on the learned behavior patterns, thereby making it possible to monitor the user's behavior and emotional state in real time and optimize the environment based on the results.

[0389] "Sensor data" is a general term for various environmental data obtained through sensors, such as temperature, lighting, and door opening / closing status.

[0390] "Artificial intelligence" is a general term for algorithms and technologies that analyze sensor data and learn and predict user behavioral patterns and emotions.

[0391] "Behavioral patterns" refer to the tendencies and habits of a series of actions in a user's daily life.

[0392] "Action" refers to an instruction such as adjusting lighting or setting temperature that the system executes based on the user's behavioral patterns and emotion recognition results.

[0393] "Device" refers to equipment or facilities that can be controlled by the system, such as lights, thermostats, and door locks.

[0394] "Emotion" refers to the user's psychological state, which is recognized based on facial expressions, tone of voice, and biometric data.

[0395] "Recognition" refers to the process of using sensor data and artificial intelligence to identify and understand user behavior and emotions.

[0396] "Notification" refers to an alert or message sent to a caregiver or relevant third party when a specified condition is met.

[0397] "Log" refers to a record of all actions and events performed by the system, which can be used for later analysis and troubleshooting.

[0398] The present invention is a smart home system targeted at dementia patients, which is capable of recognizing the user's behavioral patterns and emotions and automatically adjusting the living environment based on them.

[0399] Explaining program processing in natural language

[0400] The system is implemented using the following hardware and software.

[0401] Hardware: Temperature sensor, light sensor, door sensor, camera, microphone

[0402] Software: Server (database, AI learning module, emotion recognition engine, notification module, log management module), terminal (smartphone, tablet)

[0403] Sensor Data Collection Module

[0404] The server periodically collects data from temperature sensors, light sensors, door sensors, cameras, and microphones to monitor the user's environment and use it for subsequent processing.

[0405] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0406] AI Learning Module

[0407] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[0408] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[0409] Action Generation Module

[0410] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[0411] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[0412] Emotion Recognition Module

[0413] The server uses an emotion engine that analyzes the user's facial expressions, tone of voice, and biometric data to recognize the user's emotions. Recognized emotions include stress, anxiety, joy, etc.

[0414] Example: If a user repeatedly engages in stressful behavior, the server recognizes the pattern through an emotion engine and generates corresponding actions.

[0415] Notification Module

[0416] The server sends alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[0417] Example: If the emotion engine recognizes that the user is in a stressful state, the server sends a notification to the terminal to notify the caregiver.

[0418] Log Management Module

[0419] The server logs all actions, sensor data, and emotion recognition results, making them available for later analysis and troubleshooting.

[0420] Example: A server logs details such as which door a user opens at what time, which lights are turned on at what time, and how the user's emotional state changes, and stores this information for future reference.

[0421] Specific example details

[0422] 1. Nighttime scenario

[0423] Consider a situation where a user wakes up at 2am to go to the bathroom.

[0424] Step 1:

[0425] The server receives door opening / closing data from the door sensor at 2 a.m.

[0426] Step 2:

[0427] The server instantly switches on the lights smoothly, allowing the user to move safely.

[0428] Step 3:

[0429] The server obtains temperature data from the temperature sensor in the room and sends instructions to adjust the temperature to the appropriate level.

[0430] Step 4:

[0431] The server uses an emotion recognition module to check whether the user's emotions change during the process of going to the toilet.

[0432] Step 5:

[0433] If the user is feeling stressed or anxious, the server notifies the caregiver's terminal of that information.

[0434] Step 6:

[0435] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[0436] Step 7:

[0437] The server logs every step in detail and stores it for later analysis.

[0438] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[0439] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0440] Step 1:

[0441] The server periodically collects data from temperature sensors, light sensors, door sensors, cameras, and microphones.

[0442] Input: Real-time data from various sensors

[0443] Data processing: Taking the raw data sent from the sensors and storing it in a database

[0444] Output: The database contains the latest environmental information and user movements.

[0445] Specific operation: Every second, the server collects data on temperature, lighting status, door opening / closing status, and user image and voice data.

[0446] Step 2:

[0447] The server pre-processes the collected sensor data.

[0448] Input: Sensor data stored in a database

[0449] Data processing: removing outliers, normalizing data, and aggregating data by time

[0450] Output: Outlier-corrected and standardized data

[0451] Specific operation: The server filters out outliers from the acquired data and normalizes it. For example, if a temperature sensor shows an extreme value, it will exclude that data.

[0452] Step 3:

[0453] The server runs an AI learning module using the preprocessed data.

[0454] Input: Preprocessed sensor data

[0455] Data calculation: Input data into machine learning models to analyze user behavior patterns

[0456] Output: Learned behavioral patterns

[0457] How it works: The server uses deep learning algorithms to learn user behavior patterns based on data from the past week. For example, it uses Tensorflow (registered trademark) or PyTorch.

[0458] Step 4:

[0459] The server uses the learned model to predict user behavior.

[0460] Input: trained model, current sensor data

[0461] Data arithmetic: Predicting the most likely actions based on current data

[0462] Output: Predicted behavior pattern

[0463] Specific operation: The server inputs current data into the trained model and predicts, for example, that "the user will move to the bedroom at 10 p.m."

[0464] Step 5:

[0465] The server recognizes emotions by analyzing the user's facial expressions, tone of voice, and biometric data.

[0466] Input: Data from camera, microphone, and biometric sensors

[0467] Data Computing: Sentiment Analysis with Emotion Recognition Algorithms

[0468] Output: Perceived emotional state (e.g., stress, anxiety, joy)

[0469] Specific operation: The server uses image processing and voice analysis technology (e.g., Emotion AI) to analyze the user's emotions in real time.

[0470] Step 6:

[0471] The server generates appropriate actions based on the predicted behavior and the recognized emotions.

[0472] Input: predicted behavioral patterns, emotion recognition results

[0473] Data calculations: Determine optimal actions based on behavioral patterns and emotional states

[0474] Output: Specific action instructions (e.g., adjust lighting, set temperature, play music)

[0475] Specific operation: If the user moves to the bedroom feeling stressed, the server issues an instruction to play soft lighting and relaxing music.

[0476] Step 7:

[0477] The server sends notifications to the caregiver's terminal when certain conditions are met.

[0478] Input: Emotion recognition results, occurrence of specific events

[0479] Data calculation: Check notification conditions

[0480] Output: Notification message to caregiver device

[0481] Specific operation: When a user is judged to be under "high stress," the server immediately sends a push notification to the caregiver's smartphone to inform them of the situation.

[0482] Step 8:

[0483] The server records and stores all actions, sensor data, and emotion recognition results as logs.

[0484] Input: Output data for each step

[0485] Data processing: Record data with time stamps

[0486] Output: Detailed log data

[0487] Specific operation: The server records various data (e.g., the room temperature is 24 degrees at 10 p.m., the lights are on, the user's stress level is 50%, etc.) and stores it in secure data storage.

[0488] (Application example 2)

[0489] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0490] While existing smart home systems focus on automating the behavioral patterns and environmental settings of dementia patients, they lack the ability to respond quickly and appropriately to emotional changes. Furthermore, in brick-and-mortar stores, it is difficult for employees to instantly grasp a customer's emotional state and respond appropriately, creating challenges in improving customer satisfaction. The present invention aims to solve these challenges by providing a smart system that provides the optimal environment and customer service according to the user's emotional state.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting sensor data, means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, means for generating an action appropriate for the user based on the learned behavioral patterns, means for controlling a device to execute the generated action, means for notifying a caregiver when a specific event occurs, means for recording all actions and events as a log, means for recognizing the user's emotions, means for adjusting actions based on the user's emotional state, and means for sending a notification regarding the user's emotional state to an employee in a physical store. This enables optimal environment settings and customer service responses according to the user's emotional state.

[0492] "Sensor data" is data used to measure environmental information such as temperature, lighting, and door opening and closing status.

[0493] "Artificial intelligence" is a technology that learns and analyzes collected data to predict user behavior patterns.

[0494] "Behavior patterns" are data that indicate the tendency of a user to repeat certain time periods or actions in their daily lives.

[0495] An "action" is an instruction for controlling environment settings or devices based on the user's behavioral patterns or emotional state.

[0496] "Devices" are appliances used in homes and brick-and-mortar stores, such as lighting, heating, cooling, and security systems.

[0497] "Emotion recognition" is the process of determining a user's emotional state by analyzing their facial expressions, tone of voice, and biometric data.

[0498] A "specific event" is an important occurrence that should be dealt with by a caregiver or employee, such as abnormal behavior or emotional changes of the user.

[0499] "Notification" refers to sending alerts or messages to caregivers or employees when certain events occur.

[0500] A "log" is data that records all actions performed by the system, sensor data, and emotion recognition results.

[0501] "Employees" are staff who deal with customers in physical stores.

[0502] A "physical store" is a facility that has a physical store space and provides services and products to customers.

[0503] A "smart display" is a device that displays information in real time, allowing employees to take appropriate action.

[0504] This invention is a smart system that provides optimal environment settings and appropriate customer service based on the user's sensor data and emotional state. This system mainly consists of a sensor data collection module, an AI learning module, an action generation module, an emotion recognition module, a notification module, and a log management module.

[0505] The server periodically collects data from temperature sensors, lighting sensors, door sensors, etc., and uses this data to monitor the user's environment. The collected sensor data is analyzed by an AI learning module to learn the user's behavioral patterns and preferences. This allows the system to generate actions, such as automatically turning on appropriate lights when the user goes to the bathroom at night.

[0506] The server also uses an emotion recognition module to analyze the user's facial expressions, tone of voice, and biometric data to recognize their emotional state. Based on the recognized emotion, the server can provide an environment that helps the user relax, or in the case of a physical store, provide appropriate customer service advice to employees.

[0507] For example, if a user in a physical store has a dissatisfied expression on their face, the emotion recognition module will detect this and the server will send a notification to the employee's smart display saying, "The customer looks dissatisfied. They may need help." The employee who receives this notification can respond to the customer promptly and appropriately, improving customer satisfaction.

[0508] The server logs the various actions and events generated for future analysis and troubleshooting, allowing for continuous improvement of the system's accuracy and efficiency.

[0509] An example of a prompt sentence is, "A customer who is currently in a low mood seems to be having trouble with the product section. What is the recommended way to respond?", which is sent to the generative AI model. Based on this prompt, the AI ​​model generates advice such as "Ask the customer kindly" or "Provide additional explanation" and provides it to employees.

[0510] In this way, the present invention provides an advanced smart system that recognizes the user's emotional state in real time and then performs appropriate environmental settings and customer responses based on that recognition.

[0511] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0512] Step 1:

[0513] The server periodically collects various sensor data, such as temperature, lighting, and door sensors. This data is acquired by the sensor data collection module. The input is temperature, lighting, and door open / close status data, and the output is a collection of sensor data.

[0514] Step 2:

[0515] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences. The input is a collection of sensor data, and the output is the learning results that show the user's behavioral patterns. This makes it possible to predict what actions the user will take at a certain time.

[0516] Step 3:

[0517] The server uses an action generation module to generate optimal actions for the user based on the learned behavioral patterns. The input is the learned behavioral patterns, and the output is specific instructions or actions for the user. These actions include turning on the lights or adjusting the temperature.

[0518] Step 4:

[0519] The server uses an emotion recognition module to analyze the user's facial expressions, tone of voice, and biometric data to recognize emotions. The input is camera footage and audio data, and the output is data indicating the user's emotional state. Specifically, the camera footage is processed with a facial recognition algorithm to classify emotions.

[0520] Step 5:

[0521] The server adjusts the actions generated by the action generation module based on the recognized emotional state. The input is the emotion recognition data and the generated action, and the output is the action adjusted according to the emotion. This allows the environment to be configured according to the user's emotional state.

[0522] Step 6:

[0523] In a physical store, the server sends notifications about the emotional state to the employee's device. The input is the coordinated action and emotion recognition data, and the output is the notification content. Specifically, the employee's smart display displays a message saying, "The customer appears unhappy and may need help."

[0524] Step 7:

[0525] The server logs all actions and events and stores them for future analysis and troubleshooting. The input is various action and event data, and the output is a log file. This allows for continuous improvement and reliability of the system.

[0526] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0527] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0528] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0529] [Second embodiment]

[0530] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0531] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0532] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0533] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0534] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0535] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0536] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0537] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0538] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0539] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0540] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0541] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0542] The system of the present invention is a smart home solution for dementia patients that automatically adjusts various aspects of daily life, such as lighting, temperature, and security systems, to provide a comfortable and safe living environment.

[0543] Program Overview

[0544] The system consists of the following main modules:

[0545] 1. Sensor data collection module

[0546] 2. AI Learning Module

[0547] 3. Action Generation Module

[0548] 4. Notification Module

[0549] 5. Log Management Module

[0550] Explaining program processing in natural language

[0551] Sensor Data Collection Module

[0552] The server collects real-time data from various sensors (temperature sensors, lighting sensors, door sensors, etc.) installed in the user's environment, including temperature, lighting status, and door open / close status.

[0553] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0554] AI Learning Module

[0555] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[0556] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[0557] Action Generation Module

[0558] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[0559] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[0560] Notification Module

[0561] The server can send alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[0562] Example: If a user accidentally opens an external door in the middle of the night, the server immediately sends a notification to the device to notify the caregiver.

[0563] Log Management Module

[0564] The server logs all actions and sensor data, making it available for later analysis and troubleshooting.

[0565] Example: A server records detailed data such as which door a user opens and at what time, and which lights are turned on and at what time, and stores this information for future reference.

[0566] Specific example details

[0567] 1. Nighttime scenario

[0568] Consider a situation where a user wakes up at 2am to go to the bathroom.

[0569] The server receives data from the door sensors in real time.

[0570] The server immediately switches on the lights smoothly, allowing the user to move safely.

[0571] The server also obtains temperature data from the room's temperature sensor and issues instructions to adjust the temperature to the appropriate level.

[0572] If the door remains open for a long period of time, the server determines that it is an erroneous door opening and closing and instantly sends an alert to the terminal.

[0573] The terminal notifies the caregiver of the received alert.

[0574] Finally, the server logs every step in detail and stores it for future reference.

[0575] In this way, the system reduces the burden on caregivers while ensuring the comfort and safety of the user.

[0576] The processing flow will be explained below.

[0577] Program processing flow

[0578] Sensor Data Collection Module

[0579] Step 1:

[0580] The server periodically requests data from temperature sensors, light sensors, door sensors, etc.

[0581] Step 2:

[0582] The server receives the real-time data sent from each sensor.

[0583] Step 3:

[0584] The server stores the received data in a database.

[0585] AI Learning Module

[0586] Step 1:

[0587] The server retrieves historical sensor data from the database and performs preprocessing, which includes imputing missing values ​​and normalizing the data.

[0588] Step 2:

[0589] The server runs machine learning algorithms on the prepared data to learn user behavior patterns.

[0590] Step 3:

[0591] The server updates the obtained model and accumulates behavioral patterns.

[0592] Action Generation Module

[0593] Step 1:

[0594] The server assesses the user's current state based on the sensor data collected in real time, for example, determining whether the user is asleep or awake.

[0595] Step 2:

[0596] The server generates appropriate actions (e.g., adjusting the lights or changing the temperature) based on the learned behavioral patterns.

[0597] Step 3:

[0598] The server sends instructions to the corresponding devices to execute the generated actions.

[0599] Notification Module

[0600] Step 1:

[0601] The server monitors whether a particular event (e.g., a suspicious door opening or closing) has occurred.

[0602] Step 2:

[0603] The server generates an alert when an event occurs that meets the set conditions.

[0604] Step 3:

[0605] The server transmits the generated alert to the caregiver's terminal.

[0606] Step 4:

[0607] The device notifies the caregiver of the received alert so that they can understand the situation.

[0608] Log Management Module

[0609] Step 1:

[0610] The server generates a log of all actions and events.

[0611] Step 2:

[0612] The server stores the generated logs in a database with a timestamp.

[0613] Step 3:

[0614] The server periodically analyzes logs to monitor system performance, detect abnormalities, and identify areas for improvement.

[0615] Specific example details

[0616] For nighttime scenarios:

[0617] 1. The user wakes up at 2am to go to the bathroom.

[0618] Step 1:

[0619] The server receives door opening / closing data from the door sensor at 2 a.m.

[0620] Step 2:

[0621] The server stores the data in a database and begins processing it in real time.

[0622] Step 3:

[0623] The server recognizes that the user is moving from the bedroom to the bathroom and immediately sends a command to the lighting device to turn on the lights.

[0624] Step 4:

[0625] The server simultaneously obtains temperature data from the room's temperature sensor and sends instructions to the heating or cooling device to adjust the temperature to the desired level.

[0626] Step 5:

[0627] The server determines that the door has been opened or closed incorrectly and quickly sends an alert to the terminal.

[0628] Step 6:

[0629] The device notifies the caregiver of the received alert so that they can understand the situation.

[0630] Step 7:

[0631] The server logs every step in detail and stores it for future analysis.

[0632] This system ensures the comfort and safety of the user while reducing the burden on caregivers.

[0633] Example 1

[0634] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0635] Maintaining a comfortable and safe living environment for dementia patients places a heavy burden on caregivers. There is a need for an efficient system that can predict users' behavioral patterns and automatically generate and execute appropriate actions. The goal of this system is to reduce the burden on caregivers while ensuring the safety and comfort of users.

[0636] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0637] In this invention, the server includes means for collecting data from multiple sensors installed in the environment, means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, means for generating actions appropriate for the user based on the learned behavioral patterns, means for controlling devices to execute the generated actions, means for notifying a caregiver when a specific event occurs, and means for recording all actions and events as a log. This makes it possible to predict the user's behavior and take appropriate measures to ensure the user's safety and comfort and reduce the burden on the caregiver.

[0638] "Sensor data" is a general term for various data such as temperature, lighting, and door opening / closing status obtained from sensors installed in the environment.

[0639] "Artificial intelligence" is a technology that allows computers to imitate human intelligence, learn, reason, and assist in problem-solving.

[0640] A "behavior pattern" is a collection of data that indicates the tendencies and regularities of the user's daily behavior.

[0641] An "action" is a response or operation that the system automatically generates and executes based on the user's behavioral patterns.

[0642] A "device" is a hardware device controlled by the system, and specifically includes lighting devices, temperature control devices, door lock devices, etc.

[0643] An "event" refers to a specific action or change in state of the user, such as opening and closing a door, a sudden change in temperature, or movement during the night.

[0644] "Notifications" are alerts or messages sent to caregivers when certain events occur.

[0645] "Log" means detailed historical data of all actions and events recorded by the system for later analysis and troubleshooting.

[0646] A "caregiver" is a person whose role is to support the lives of people with dementia and provide the necessary care.

[0647] "Environment" refers to the residence where the user lives and the surrounding area, and refers to the space where the sensor is installed.

[0648] This invention relates to a smart home solution for dementia patients. The system automatically adjusts various aspects of daily life such as lighting, temperature, and security systems to provide a comfortable and safe living environment.

[0649] Program Overview

[0650] The system includes the following main modules:

[0651] 1. Sensor data collection module

[0652] 2. AI Learning Module

[0653] 3. Action Generation Module

[0654] 4. Notification Module

[0655] 5. Log Management Module

[0656] Sensor Data Collection Module

[0657] Users install temperature sensors, lighting sensors, door sensors, etc. in their living spaces. These sensors collect the user's environmental data (temperature, lighting status, door open / close status, etc.) in real time. This sensor data is sent to a server and recorded.

[0658] As a specific example, when a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0659] AI Learning Module

[0660] The server uses collected sensor data to run a learning module using artificial intelligence (AI) to learn the user's behavioral patterns and environmental preferences, making it possible to predict the user's behavior at specific times and in specific situations.

[0661] As a specific example, a server that learns that a user moves to their bedroom around 10 p.m. every night will set the room temperature to be adjusted appropriately during that time.

[0662] Action Generation Module

[0663] The server generates appropriate actions based on the behavioral patterns obtained from the AI ​​learning module, such as automatically turning on the lights when the user wakes up.

[0664] As a specific example, the server learns that the user wakes up at 7:00 every morning and sets the lights to gradually brighten accordingly.

[0665] Notification Module

[0666] The server sends alerts and notifications to the caregiver's device when certain events occur, allowing the caregiver to understand the user's real-time situation and respond quickly.

[0667] As a specific example, if a user accidentally opens an external door in the middle of the night, the server quickly detects this information and sends a notification to the caregiver's terminal.

[0668] Log Management Module

[0669] The server records detailed logs of all actions and events, which can be used for later analysis and troubleshooting.

[0670] For example, the server records detailed data such as which door the user opened at what time, and which lights were turned on at what time, and stores this information for future reference.

[0671] Specific example details

[0672] 1. Nighttime scenario

[0673] If the user wakes up at 2am to go to the bathroom

[0674] The server receives data from the door sensors in real time.

[0675] The server instantly switches on the lights smoothly to allow the user to move safely.

[0676] The server obtains temperature data from the temperature sensor in the room and issues instructions to adjust the temperature to the appropriate level.

[0677] If the door is left open for a long period of time, the server determines that it is an erroneous door opening and closing and instantly sends an alert to the caregiver's device.

[0678] The device notifies the caregiver of the received alert.

[0679] Finally, the server logs every step in detail and stores it for future reference.

[0680] Examples of prompt statements

[0681] Below are some example prompts to input to a generative AI model:

[0682] "Describe a smart home system that automatically turns on the lights, maintains the appropriate room temperature, and notifies caregivers as needed when a user opens the door to go to the bathroom at night."

[0683] In this way, the present invention aims to reduce the burden on caregivers while ensuring the comfort and safety of the user.

[0684] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0685] Specific processing flow of the program

[0686] Sensor Data Collection Module

[0687] Step 1:

[0688] Users install temperature sensors, light sensors, door sensors, etc. in their living spaces to monitor the user's behavior and environmental conditions in real time.

[0689] Input: Sensor installation area information

[0690] Operation: Sensor device installation and initial setup

[0691] Step 2:

[0692] The server collects real-time data from each sensor, including the room temperature, lighting status, and door open / close status.

[0693] Input: Real-time data from each sensor

[0694] What it does: polls and collects data

[0695] Output: Real-time sensor data

[0696] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0697] AI Learning Module

[0698] Step 3:

[0699] The server runs a learning module using artificial intelligence (AI) based on the collected sensor data. The AI ​​module learns the user's behavioral patterns and makes it possible to predict the user's behavior at specific times and in specific situations.

[0700] Input: Real-time sensor data

[0701] What it does: Analyze data and run learning algorithms

[0702] Output: Learned behavioral patterns

[0703] Example: AI learns the pattern of a user moving to the bedroom around 10 p.m. each night and adjusts the room temperature appropriately at that time.

[0704] Action Generation Module

[0705] Step 4:

[0706] The server generates optimal actions for users based on behavioral patterns obtained from the AI ​​learning module, such as automatically turning on lights or adjusting the temperature.

[0707] Input: Learned behavioral patterns

[0708] Behavior: Condition determination and action generation

[0709] Output: The generated actions

[0710] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[0711] Notification Module

[0712] Step 5:

[0713] The server sends alerts and notifications to the caregiver's device when certain events occur, allowing the caregiver to understand the user's current condition in real time and respond quickly to emergencies.

[0714] Input: A specific event (e.g., door opening and closing at midnight)

[0715] Action: Detecting an event and sending a notification

[0716] Output: Notification sent to caregiver's device

[0717] Example: If a user accidentally opens an external door late at night, the server detects this information and promptly sends a notification to the caregiver's device.

[0718] Log Management Module

[0719] Step 6:

[0720] The server records detailed logs of all actions and events, which are stored for later analysis and troubleshooting.

[0721] Input: Generated actions and detected events

[0722] Operation: Recording and storing data

[0723] Output: Detailed log data

[0724] Example: A server records detailed data such as which door a user opens and at what time, and which lights are turned on and at what time, and stores this information for future reference.

[0725] Thus, each processing step of this program aims to reduce the burden on caregivers while ensuring the comfort and safety of the user.

[0726] (Application example 1)

[0727] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0728] Current physical store management systems lack the ability to automatically adjust environmental factors such as temperature and lighting according to customer visits, resulting in an inconsistent customer experience and a heavy burden on managers to make manual adjustments. Notification functions for abnormalities are also limited, making it difficult to respond quickly. This hinders efficient store operations and improved customer satisfaction.

[0729] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0730] In this invention, the server includes means for collecting sensor data, means for learning user behavior patterns based on the collected sensor data using artificial intelligence, means for generating actions appropriate for the user based on the learned behavior patterns, means for controlling devices to execute the generated actions, means for notifying an administrator when a specific event occurs, means for recording all actions and events as a log, and means for automatically adjusting environmental factors (temperature, lighting, etc.) in the physical store according to customer behavior patterns, thereby enabling effective environmental management and rapid response to abnormalities.

[0731] "Sensor data" refers to environmental information collected by sensors, such as temperature, lighting, human presence, and door opening and closing status.

[0732] "Artificial intelligence" is the technology that enables computer systems to analyze data patterns, learn, and make predictions.

[0733] A "behavioral pattern" is a collection of data that indicates the tendency of regular behavior of a user or customer.

[0734] "Action" refers to optimal behavior or adjustments generated based on learned behavioral patterns.

[0735] "Device" is a general term for equipment or facilities that are controlled to execute the generated action.

[0736] "Administrator" refers to the person in charge of running the system or store, and who will be notified of events.

[0737] A "log" is a data record that records all actions and events and is used for later analysis and reference.

[0738] The system of the present invention is a smart environment management system that aims to improve the customer experience in physical stores and streamline store management. This system is composed of the following main modules.

[0739] 1. Sensor data collection module

[0740] The server collects data in real time from temperature sensors, lighting sensors, motion sensors, door sensors, etc. installed in the physical store, thereby obtaining environmental information within the store (temperature, lighting, human movement, door opening / closing status).

[0741] 2. AI Learning Module

[0742] The server uses artificial intelligence to learn customer behavior patterns based on collected sensor data. This module uses data analysis and pattern recognition to predict customer movements within the store at specific times.

[0743] 3. Action Generation Module

[0744] The server generates optimal actions based on learned behavioral patterns, such as automatically brightening the lights and maintaining an appropriate temperature when the store is crowded.

[0745] 4. Device Control Module

[0746] The server controls devices (such as lighting equipment, air conditioning equipment, and security devices) to execute the generated actions, thereby providing a comfortable environment for the customer.

[0747] 5. Notification Module

[0748] The server sends an alert to the administrator's device when a specific event occurs. For example, if it detects abnormal activity, it will immediately notify the administrator.

[0749] 6. Log Management Module

[0750] The server records all actions and events in detail and makes them available for later analysis and troubleshooting.

[0751] Hardware and software used

[0752] Hardware: Temperature sensors, lighting sensors, motion sensors, door sensors, lighting equipment, air conditioning equipment, security devices

[0753] Software: Python, data collection API, notification API, log storage function

[0754] Specific examples

[0755] For example, consider the process when a particular area in a store is crowded.

[0756] The server receives congestion data from the motion sensors in real time.

[0757] The server uses an AI learning module to analyze congestion patterns.

[0758] The server uses an action generation module to generate actions to brighten the lights and lower the temperature.

[0759] The server controls lighting fixtures and air conditioning equipment through device control modules.

[0760] As a result of this series of actions, customers can visit the store in a more comfortable environment.

[0761] Prompt Sentence Examples

[0762] For example, by inputting the following prompt sentence into the generative AI model, it is possible to generate a document that improves the action generation algorithm.

[0763] "Please provide a detailed description of appropriate environmental adjustment methods when stores are crowded, based on data obtained from motion sensors and temperature sensors in the physical store."

[0764] Based on this prompt, the generative AI can suggest specific actions and adjustments.

[0765] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0766] Step 1:

[0767] The server collects data in real time from temperature sensors, lighting sensors, motion sensors, door sensors, and other sensors installed in the physical store. Environmental information from each sensor (temperature, lighting, human movement, door opening / closing status) is taken in as input, and sensor data is obtained as output. This sensor data makes it possible to grasp the current situation inside the store.

[0768] Step 2:

[0769] The server uses artificial intelligence (AI) to learn customer behavior patterns based on the collected sensor data. The sensor data is passed as input to an AI learning module, which performs data analysis and pattern recognition. The output includes customer behavior patterns and trends, making it possible to predict customer behavior at specific times and in specific situations.

[0770] Step 3:

[0771] The server generates optimal actions based on the learned behavioral patterns. It uses the behavioral patterns obtained from the AI ​​learning module and real-time sensor data as input. It performs data calculations (adjusting temperature, lighting, etc.) and obtains specific actions as output. These actions include automatically adjusting the temperature and lighting in the store.

[0772] Step 4:

[0773] The server controls devices (such as lighting equipment, air conditioning equipment, and security devices) to execute the generated actions. It receives action instructions from the action generation module as input and sends specific signals to control each device as output. This operation makes the in-store environment more comfortable for customers.

[0774] Step 5:

[0775] The server sends an alert to the administrator's terminal when a specific event occurs. It monitors sensor data and system status as input, and generates notification data as output when an abnormality or specific condition is detected, which is sent to the administrator's terminal. This allows the administrator to quickly respond to the abnormality.

[0776] Step 6:

[0777] The server records all actions and events in detail and saves them as log data. It collects sensor data, action data, notification data, etc. as input and records them in a log file as output. This log data can be used later for analysis and troubleshooting.

[0778] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0779] The system of the present invention combines a smart home solution for dementia patients with an emotion engine that recognizes the user's emotions. The system automatically adjusts various aspects of daily life, such as lighting, temperature, and security systems, to provide a comfortable and safe living environment based on the user's emotional state.

[0780] Program Overview

[0781] The system consists of the following main modules:

[0782] 1. Sensor data collection module

[0783] 2. AI Learning Module

[0784] 3. Action Generation Module

[0785] 4. Emotion Recognition Module

[0786] 5. Notification Module

[0787] 6. Log Management Module

[0788] Explaining program processing in natural language

[0789] Sensor Data Collection Module

[0790] The server periodically collects data from temperature sensors, light sensors, door sensors, etc. This data is used to monitor the user's environment and for subsequent processing.

[0791] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0792] AI Learning Module

[0793] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[0794] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[0795] Action Generation Module

[0796] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[0797] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[0798] Emotion Recognition Module

[0799] The server uses an emotion engine that analyzes the user's facial expressions, tone of voice, and biometric data to recognize the user's emotions. Recognized emotions include stress, anxiety, joy, etc.

[0800] Example: If a user repeatedly engages in stressful behavior, the server recognizes the pattern through an emotion engine and generates corresponding actions.

[0801] Notification Module

[0802] The server sends alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[0803] Example: If the emotion engine recognizes that the user is in a stressful state, the server sends a notification to the terminal to notify the caregiver.

[0804] Log Management Module

[0805] The server logs all actions, sensor data, and emotion recognition results, making them available for later analysis and troubleshooting.

[0806] Example: A server logs details such as which door a user opens at what time, which lights are turned on at what time, and how the user's emotional state changes, and stores this information for future reference.

[0807] Specific example details

[0808] 1. Nighttime scenario

[0809] Consider a situation where a user wakes up at 2am to go to the bathroom.

[0810] Step 1:

[0811] The server receives door opening / closing data from the door sensor at 2 a.m.

[0812] Step 2:

[0813] The server instantly switches on the lights smoothly, allowing the user to move safely.

[0814] Step 3:

[0815] The server obtains temperature data from the temperature sensor in the room and sends instructions to adjust the temperature to the appropriate level.

[0816] Step 4:

[0817] The server uses an emotion recognition module to check whether the user's emotions change during the process of going to the toilet.

[0818] Step 5:

[0819] If the user is feeling stressed or anxious, the server notifies the caregiver's terminal of that information.

[0820] Step 6:

[0821] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[0822] Step 7:

[0823] The server logs every step in detail and stores it for later analysis.

[0824] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[0825] The processing flow will be explained below.

[0826] Program processing flow

[0827] Sensor Data Collection Module

[0828] Step 1:

[0829] The server periodically requests data from temperature sensors, light sensors, and door sensors.

[0830] Step 2:

[0831] The server receives the real-time data sent from each sensor.

[0832] Step 3:

[0833] The server stores the received data in a database.

[0834] AI Learning Module

[0835] Step 1:

[0836] The server retrieves historical sensor data from the database and performs preprocessing, which includes imputing missing values ​​and normalizing the data.

[0837] Step 2:

[0838] The server runs machine learning algorithms on the prepared data to learn user behavior patterns.

[0839] Step 3:

[0840] The server updates the obtained model and accumulates behavioral patterns.

[0841] Action Generation Module

[0842] Step 1:

[0843] The server assesses the user's current state based on the sensor data collected in real time, for example, determining whether the user is asleep or awake.

[0844] Step 2:

[0845] The server generates appropriate actions (e.g., adjusting the lights or changing the temperature) based on the learned behavioral patterns.

[0846] Step 3:

[0847] The server sends instructions to the corresponding devices to execute the generated actions.

[0848] Emotion Recognition Module

[0849] Step 1:

[0850] The server collects the user's facial expressions, tone of voice, and biometric data (e.g., heart rate and skin potential) in real time.

[0851] Step 2:

[0852] The server analyzes the collected data through an emotion engine to recognize the user's emotion.

[0853] Step 3:

[0854] The server modifies or corrects system actions and notifications based on the recognized emotions.

[0855] Notification Module

[0856] Step 1:

[0857] The server monitors whether a specific event has occurred (for example, a suspicious door opening or the user's state of stress).

[0858] Step 2:

[0859] The server generates an alert when an event that meets the condition occurs.

[0860] Step 3:

[0861] The server transmits the generated alert to the caregiver's terminal.

[0862] Step 4:

[0863] The device notifies the caregiver of the received alert so that they can understand the situation.

[0864] Log Management Module

[0865] Step 1:

[0866] The server generates logs of all actions, events, and emotion recognition results.

[0867] Step 2:

[0868] The server stores the generated logs in a database with a timestamp.

[0869] Step 3:

[0870] The server periodically analyzes logs to monitor system performance, detect abnormalities, and identify areas for improvement.

[0871] Specific example details

[0872] 1. Nighttime scenario

[0873] The user wakes up at 2am to go to the bathroom.

[0874] Step 1:

[0875] The server receives door opening / closing data from the door sensor at 2 a.m.

[0876] Step 2:

[0877] The server stores the data in a database and begins processing it in real time.

[0878] Step 3:

[0879] The server recognizes that the user is moving from the bedroom to the bathroom and immediately sends a command to the lighting device to turn on the lights.

[0880] Step 4:

[0881] At the same time, the server obtains temperature data from the room's temperature sensor and sends instructions to the heating or cooling device to adjust the temperature to the appropriate level.

[0882] Step 5:

[0883] The server collects the user's facial expressions, tone of voice, and biometric data through an emotion recognition module to recognize the user's emotions.

[0884] Step 6:

[0885] If the user feels stress or anxiety while going to the toilet, the server notifies the caregiver's terminal of that information in real time.

[0886] Step 7:

[0887] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[0888] Step 8:

[0889] The server logs every step in detail and stores it for later analysis.

[0890] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[0891] Example 2

[0892] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0893] In modern times, smart home systems for dementia patients are important for supporting their daily lives. However, conventional smart home systems have limited automation capabilities based on the user's behavioral patterns and lack the ability to respond to the user's emotional changes. Therefore, there is a need for a system that can monitor the emotional state of dementia patients in real time and adjust the environment accordingly.

[0894] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0895] In this invention, the server includes means for collecting sensor data, means for learning user behavior patterns based on the collected sensor data using artificial intelligence, and means for generating actions suitable for the user based on the learned behavior patterns, thereby making it possible to monitor the user's behavior and emotional state in real time and optimize the environment based on the results.

[0896] "Sensor data" is a general term for various environmental data obtained through sensors, such as temperature, lighting, and door opening / closing status.

[0897] "Artificial intelligence" is a general term for algorithms and technologies that analyze sensor data and learn and predict user behavioral patterns and emotions.

[0898] "Behavioral patterns" refer to the tendencies and habits of a series of actions in a user's daily life.

[0899] "Action" refers to an instruction such as adjusting lighting or setting temperature that the system executes based on the user's behavioral patterns and emotion recognition results.

[0900] "Device" refers to equipment or facilities that can be controlled by the system, such as lights, thermostats, and door locks.

[0901] "Emotion" refers to the user's psychological state, which is recognized based on facial expressions, tone of voice, and biometric data.

[0902] "Recognition" refers to the process of using sensor data and artificial intelligence to identify and understand user behavior and emotions.

[0903] "Notification" refers to an alert or message sent to a caregiver or relevant third party when a specified condition is met.

[0904] "Log" refers to a record of all actions and events performed by the system, which can be used for later analysis and troubleshooting.

[0905] The present invention is a smart home system targeted at dementia patients, which is capable of recognizing the user's behavioral patterns and emotions and automatically adjusting the living environment based on them.

[0906] Explaining program processing in natural language

[0907] The system is implemented using the following hardware and software.

[0908] Hardware: Temperature sensor, light sensor, door sensor, camera, microphone

[0909] Software: Server (database, AI learning module, emotion recognition engine, notification module, log management module), terminal (smartphone, tablet)

[0910] Sensor Data Collection Module

[0911] The server periodically collects data from temperature sensors, light sensors, door sensors, cameras, and microphones to monitor the user's environment and use it for subsequent processing.

[0912] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[0913] AI Learning Module

[0914] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[0915] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[0916] Action Generation Module

[0917] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[0918] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[0919] Emotion Recognition Module

[0920] The server uses an emotion engine that analyzes the user's facial expressions, tone of voice, and biometric data to recognize the user's emotions. Recognized emotions include stress, anxiety, joy, etc.

[0921] Example: If a user repeatedly engages in stressful behavior, the server recognizes the pattern through an emotion engine and generates corresponding actions.

[0922] Notification Module

[0923] The server sends alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[0924] Example: If the emotion engine recognizes that the user is in a stressful state, the server sends a notification to the terminal to notify the caregiver.

[0925] Log Management Module

[0926] The server logs all actions, sensor data, and emotion recognition results, making them available for later analysis and troubleshooting.

[0927] Example: A server logs details such as which door a user opens at what time, which lights are turned on at what time, and how the user's emotional state changes, and stores this information for future reference.

[0928] Specific example details

[0929] 1. Nighttime scenario

[0930] Consider a situation where a user wakes up at 2am to go to the bathroom.

[0931] Step 1:

[0932] The server receives door opening / closing data from the door sensor at 2 a.m.

[0933] Step 2:

[0934] The server instantly switches on the lights smoothly, allowing the user to move safely.

[0935] Step 3:

[0936] The server obtains temperature data from the temperature sensor in the room and sends instructions to adjust the temperature to the appropriate level.

[0937] Step 4:

[0938] The server uses an emotion recognition module to check whether the user's emotions change during the process of going to the toilet.

[0939] Step 5:

[0940] If the user is feeling stressed or anxious, the server notifies the caregiver's terminal of that information.

[0941] Step 6:

[0942] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[0943] Step 7:

[0944] The server logs every step in detail and stores it for later analysis.

[0945] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[0946] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0947] Step 1:

[0948] The server periodically collects data from temperature sensors, light sensors, door sensors, cameras, and microphones.

[0949] Input: Real-time data from various sensors

[0950] Data processing: Taking the raw data sent from the sensors and storing it in a database

[0951] Output: The database contains the latest environmental information and user movements.

[0952] Specific operation: Every second, the server collects data on temperature, lighting status, door opening / closing status, and user image and voice data.

[0953] Step 2:

[0954] The server pre-processes the collected sensor data.

[0955] Input: Sensor data stored in a database

[0956] Data processing: removing outliers, normalizing data, and aggregating data by time

[0957] Output: Outlier-corrected and standardized data

[0958] Specific operation: The server filters out outliers from the acquired data and normalizes it. For example, if a temperature sensor shows an extreme value, it will exclude that data.

[0959] Step 3:

[0960] The server runs an AI learning module using the preprocessed data.

[0961] Input: Preprocessed sensor data

[0962] Data calculation: Input data into machine learning models to analyze user behavior patterns

[0963] Output: Learned behavioral patterns

[0964] How it works: The server uses deep learning algorithms to learn user behavior patterns based on data from the past week. For example, it uses TensorFlow or PyTorch.

[0965] Step 4:

[0966] The server uses the learned model to predict user behavior.

[0967] Input: trained model, current sensor data

[0968] Data arithmetic: Predicting the most likely actions based on current data

[0969] Output: Predicted behavior pattern

[0970] Specific operation: The server inputs current data into the trained model and predicts, for example, that "the user will move to the bedroom at 10 p.m."

[0971] Step 5:

[0972] The server recognizes emotions by analyzing the user's facial expressions, tone of voice, and biometric data.

[0973] Input: Data from camera, microphone, and biometric sensors

[0974] Data Computing: Sentiment Analysis with Emotion Recognition Algorithms

[0975] Output: Perceived emotional state (e.g., stress, anxiety, joy)

[0976] Specific operation: The server uses image processing and voice analysis technology (e.g., Emotion AI) to analyze the user's emotions in real time.

[0977] Step 6:

[0978] The server generates appropriate actions based on the predicted behavior and the recognized emotions.

[0979] Input: predicted behavioral patterns, emotion recognition results

[0980] Data calculations: Determine optimal actions based on behavioral patterns and emotional states

[0981] Output: Specific action instructions (e.g., adjust lighting, set temperature, play music)

[0982] Specific operation: If the user moves to the bedroom feeling stressed, the server issues an instruction to play soft lighting and relaxing music.

[0983] Step 7:

[0984] The server sends notifications to the caregiver's terminal when certain conditions are met.

[0985] Input: Emotion recognition results, occurrence of specific events

[0986] Data calculation: Check notification conditions

[0987] Output: Notification message to caregiver device

[0988] Specific operation: When a user is judged to be under "high stress," the server immediately sends a push notification to the caregiver's smartphone to inform them of the situation.

[0989] Step 8:

[0990] The server records and stores all actions, sensor data, and emotion recognition results as logs.

[0991] Input: Output data for each step

[0992] Data processing: Record data with time stamps

[0993] Output: Detailed log data

[0994] Specific operation: The server records various data (e.g., the room temperature is 24 degrees at 10 p.m., the lights are on, the user's stress level is 50%, etc.) and stores it in secure data storage.

[0995] (Application example 2)

[0996] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0997] While existing smart home systems focus on automating the behavioral patterns and environmental settings of dementia patients, they lack the ability to respond quickly and appropriately to emotional changes. Furthermore, in brick-and-mortar stores, it is difficult for employees to instantly grasp a customer's emotional state and respond appropriately, creating challenges in improving customer satisfaction. The present invention aims to solve these challenges by providing a smart system that provides the optimal environment and customer service according to the user's emotional state.

[0998] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting sensor data, means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, means for generating an action appropriate for the user based on the learned behavioral patterns, means for controlling a device to execute the generated action, means for notifying a caregiver when a specific event occurs, means for recording all actions and events as a log, means for recognizing the user's emotions, means for adjusting actions based on the user's emotional state, and means for sending a notification regarding the user's emotional state to an employee in a physical store. This enables optimal environment settings and customer service responses according to the user's emotional state.

[0999] "Sensor data" is data used to measure environmental information such as temperature, lighting, and door opening and closing status.

[1000] "Artificial intelligence" is a technology that learns and analyzes collected data to predict user behavior patterns.

[1001] "Behavior patterns" are data that indicate the tendency of a user to repeat certain time periods or actions in their daily lives.

[1002] An "action" is an instruction for controlling environment settings or devices based on the user's behavioral patterns or emotional state.

[1003] "Devices" are appliances used in homes and brick-and-mortar stores, such as lighting, heating, cooling, and security systems.

[1004] "Emotion recognition" is the process of determining a user's emotional state by analyzing their facial expressions, tone of voice, and biometric data.

[1005] A "specific event" is an important occurrence that should be dealt with by a caregiver or employee, such as abnormal behavior or emotional changes of the user.

[1006] "Notification" refers to sending alerts or messages to caregivers or employees when certain events occur.

[1007] A "log" is data that records all actions performed by the system, sensor data, and emotion recognition results.

[1008] "Employees" are staff who deal with customers in physical stores.

[1009] A "physical store" is a facility that has a physical store space and provides services and products to customers.

[1010] A "smart display" is a device that displays information in real time, allowing employees to take appropriate action.

[1011] This invention is a smart system that provides optimal environment settings and appropriate customer service based on the user's sensor data and emotional state. This system mainly consists of a sensor data collection module, an AI learning module, an action generation module, an emotion recognition module, a notification module, and a log management module.

[1012] The server periodically collects data from temperature sensors, lighting sensors, door sensors, etc., and uses this data to monitor the user's environment. The collected sensor data is analyzed by an AI learning module to learn the user's behavioral patterns and preferences. This allows the system to generate actions, such as automatically turning on appropriate lights when the user goes to the bathroom at night.

[1013] The server also uses an emotion recognition module to analyze the user's facial expressions, tone of voice, and biometric data to recognize their emotional state. Based on the recognized emotion, the server can provide an environment that helps the user relax, or in the case of a physical store, provide appropriate customer service advice to employees.

[1014] For example, if a user in a physical store has a dissatisfied expression on their face, the emotion recognition module will detect this and the server will send a notification to the employee's smart display saying, "The customer looks dissatisfied. They may need help." The employee who receives this notification can respond to the customer promptly and appropriately, improving customer satisfaction.

[1015] The server logs the various actions and events generated for future analysis and troubleshooting, allowing for continuous improvement of the system's accuracy and efficiency.

[1016] An example of a prompt sentence is, "A customer who is currently in a low mood seems to be having trouble with the product section. What is the recommended way to respond?", which is sent to the generative AI model. Based on this prompt, the AI ​​model generates advice such as "Ask the customer kindly" or "Provide additional explanation" and provides it to employees.

[1017] In this way, the present invention provides an advanced smart system that recognizes the user's emotional state in real time and then performs appropriate environmental settings and customer responses based on that recognition.

[1018] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1019] Step 1:

[1020] The server periodically collects various sensor data, such as temperature, lighting, and door sensors. This data is acquired by the sensor data collection module. The input is temperature, lighting, and door open / close status data, and the output is a collection of sensor data.

[1021] Step 2:

[1022] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences. The input is a collection of sensor data, and the output is the learning results that show the user's behavioral patterns. This makes it possible to predict what actions the user will take at a certain time.

[1023] Step 3:

[1024] The server uses an action generation module to generate optimal actions for the user based on the learned behavioral patterns. The input is the learned behavioral patterns, and the output is specific instructions or actions for the user. These actions include turning on the lights or adjusting the temperature.

[1025] Step 4:

[1026] The server uses an emotion recognition module to analyze the user's facial expressions, tone of voice, and biometric data to recognize emotions. The input is camera footage and audio data, and the output is data indicating the user's emotional state. Specifically, the camera footage is processed with a facial recognition algorithm to classify emotions.

[1027] Step 5:

[1028] The server adjusts the actions generated by the action generation module based on the recognized emotional state. The input is the emotion recognition data and the generated action, and the output is the action adjusted according to the emotion. This allows the environment to be configured according to the user's emotional state.

[1029] Step 6:

[1030] In a physical store, the server sends notifications about the emotional state to the employee's device. The input is the coordinated action and emotion recognition data, and the output is the notification content. Specifically, the employee's smart display displays a message saying, "The customer appears unhappy and may need help."

[1031] Step 7:

[1032] The server logs all actions and events and stores them for future analysis and troubleshooting. The input is various action and event data, and the output is a log file. This allows for continuous improvement and reliability of the system.

[1033] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1034] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1035] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1036] [Third embodiment]

[1037] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1038] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1039] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1040] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1041] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1042] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1043] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1044] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1045] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1047] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1048] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1049] The system of the present invention is a smart home solution for dementia patients that automatically adjusts various aspects of daily life, such as lighting, temperature, and security systems, to provide a comfortable and safe living environment.

[1050] Program Overview

[1051] The system consists of the following main modules:

[1052] 1. Sensor data collection module

[1053] 2. AI Learning Module

[1054] 3. Action Generation Module

[1055] 4. Notification Module

[1056] 5. Log Management Module

[1057] Explaining program processing in natural language

[1058] Sensor Data Collection Module

[1059] The server collects real-time data from various sensors (temperature sensors, lighting sensors, door sensors, etc.) installed in the user's environment, including temperature, lighting status, and door open / close status.

[1060] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1061] AI Learning Module

[1062] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[1063] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[1064] Action Generation Module

[1065] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[1066] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[1067] Notification Module

[1068] The server can send alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[1069] Example: If a user accidentally opens an external door in the middle of the night, the server immediately sends a notification to the device to notify the caregiver.

[1070] Log Management Module

[1071] The server logs all actions and sensor data, making it available for later analysis and troubleshooting.

[1072] Example: A server records detailed data such as which door a user opens and at what time, and which lights are turned on and at what time, and stores this information for future reference.

[1073] Specific example details

[1074] 1. Nighttime scenario

[1075] Consider a situation where a user wakes up at 2am to go to the bathroom.

[1076] The server receives data from the door sensors in real time.

[1077] The server immediately switches on the lights smoothly, allowing the user to move safely.

[1078] The server also obtains temperature data from the room's temperature sensor and issues instructions to adjust the temperature to the appropriate level.

[1079] If the door remains open for a long period of time, the server determines that it is an erroneous door opening and closing and instantly sends an alert to the terminal.

[1080] The terminal notifies the caregiver of the received alert.

[1081] Finally, the server logs every step in detail and stores it for future reference.

[1082] In this way, the system reduces the burden on caregivers while ensuring the comfort and safety of the user.

[1083] The processing flow will be explained below.

[1084] Program processing flow

[1085] Sensor Data Collection Module

[1086] Step 1:

[1087] The server periodically requests data from temperature sensors, light sensors, door sensors, etc.

[1088] Step 2:

[1089] The server receives the real-time data sent from each sensor.

[1090] Step 3:

[1091] The server stores the received data in a database.

[1092] AI Learning Module

[1093] Step 1:

[1094] The server retrieves historical sensor data from the database and performs preprocessing, which includes imputing missing values ​​and normalizing the data.

[1095] Step 2:

[1096] The server runs machine learning algorithms on the prepared data to learn user behavior patterns.

[1097] Step 3:

[1098] The server updates the obtained model and accumulates behavioral patterns.

[1099] Action Generation Module

[1100] Step 1:

[1101] The server assesses the user's current state based on the sensor data collected in real time, for example, determining whether the user is asleep or awake.

[1102] Step 2:

[1103] The server generates appropriate actions (e.g., adjusting the lights or changing the temperature) based on the learned behavioral patterns.

[1104] Step 3:

[1105] The server sends instructions to the corresponding devices to execute the generated actions.

[1106] Notification Module

[1107] Step 1:

[1108] The server monitors whether a particular event (e.g., a suspicious door opening or closing) has occurred.

[1109] Step 2:

[1110] The server generates an alert when an event occurs that meets the set conditions.

[1111] Step 3:

[1112] The server transmits the generated alert to the caregiver's terminal.

[1113] Step 4:

[1114] The device notifies the caregiver of the received alert so that they can understand the situation.

[1115] Log Management Module

[1116] Step 1:

[1117] The server generates a log of all actions and events.

[1118] Step 2:

[1119] The server stores the generated logs in a database with a timestamp.

[1120] Step 3:

[1121] The server periodically analyzes logs to monitor system performance, detect abnormalities, and identify areas for improvement.

[1122] Specific example details

[1123] For nighttime scenarios:

[1124] 1. The user wakes up at 2am to go to the bathroom.

[1125] Step 1:

[1126] The server receives door opening / closing data from the door sensor at 2 a.m.

[1127] Step 2:

[1128] The server stores the data in a database and begins processing it in real time.

[1129] Step 3:

[1130] The server recognizes that the user is moving from the bedroom to the bathroom and immediately sends a command to the lighting device to turn on the lights.

[1131] Step 4:

[1132] The server simultaneously obtains temperature data from the room's temperature sensor and sends instructions to the heating or cooling device to adjust the temperature to the desired level.

[1133] Step 5:

[1134] The server determines that the door has been opened or closed incorrectly and quickly sends an alert to the terminal.

[1135] Step 6:

[1136] The device notifies the caregiver of the received alert so that they can understand the situation.

[1137] Step 7:

[1138] The server logs every step in detail and stores it for future analysis.

[1139] This system ensures the comfort and safety of the user while reducing the burden on caregivers.

[1140] Example 1

[1141] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1142] Maintaining a comfortable and safe living environment for dementia patients places a heavy burden on caregivers. There is a need for an efficient system that can predict users' behavioral patterns and automatically generate and execute appropriate actions. The goal of this system is to reduce the burden on caregivers while ensuring the safety and comfort of users.

[1143] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1144] In this invention, the server includes means for collecting data from multiple sensors installed in the environment, means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, means for generating actions appropriate for the user based on the learned behavioral patterns, means for controlling devices to execute the generated actions, means for notifying a caregiver when a specific event occurs, and means for recording all actions and events as a log. This makes it possible to predict the user's behavior and take appropriate measures to ensure the user's safety and comfort and reduce the burden on the caregiver.

[1145] "Sensor data" is a general term for various data such as temperature, lighting, and door opening / closing status obtained from sensors installed in the environment.

[1146] "Artificial intelligence" is a technology that allows computers to imitate human intelligence, learn, reason, and assist in problem-solving.

[1147] A "behavior pattern" is a collection of data that indicates the tendencies and regularities of the user's daily behavior.

[1148] An "action" is a response or operation that the system automatically generates and executes based on the user's behavioral patterns.

[1149] A "device" is a hardware device controlled by the system, and specifically includes lighting devices, temperature control devices, door lock devices, etc.

[1150] An "event" refers to a specific action or change in state of the user, such as opening and closing a door, a sudden change in temperature, or movement during the night.

[1151] "Notifications" are alerts or messages sent to caregivers when certain events occur.

[1152] "Log" means detailed historical data of all actions and events recorded by the system for later analysis and troubleshooting.

[1153] A "caregiver" is a person whose role is to support the lives of people with dementia and provide the necessary care.

[1154] "Environment" refers to the residence where the user lives and the surrounding area, and refers to the space where the sensor is installed.

[1155] This invention relates to a smart home solution for dementia patients. The system automatically adjusts various aspects of daily life such as lighting, temperature, and security systems to provide a comfortable and safe living environment.

[1156] Program Overview

[1157] The system includes the following main modules:

[1158] 1. Sensor data collection module

[1159] 2. AI Learning Module

[1160] 3. Action Generation Module

[1161] 4. Notification Module

[1162] 5. Log Management Module

[1163] Sensor Data Collection Module

[1164] Users install temperature sensors, lighting sensors, door sensors, etc. in their living spaces. These sensors collect the user's environmental data (temperature, lighting status, door open / close status, etc.) in real time. This sensor data is sent to a server and recorded.

[1165] As a specific example, when a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1166] AI Learning Module

[1167] The server uses collected sensor data to run a learning module using artificial intelligence (AI) to learn the user's behavioral patterns and environmental preferences, making it possible to predict the user's behavior at specific times and in specific situations.

[1168] As a specific example, a server that learns that a user moves to their bedroom around 10 p.m. every night will set the room temperature to be adjusted appropriately during that time.

[1169] Action Generation Module

[1170] The server generates appropriate actions based on the behavioral patterns obtained from the AI ​​learning module, such as automatically turning on the lights when the user wakes up.

[1171] As a specific example, the server learns that the user wakes up at 7:00 every morning and sets the lights to gradually brighten accordingly.

[1172] Notification Module

[1173] The server sends alerts and notifications to the caregiver's device when certain events occur, allowing the caregiver to understand the user's real-time situation and respond quickly.

[1174] As a specific example, if a user accidentally opens an external door in the middle of the night, the server quickly detects this information and sends a notification to the caregiver's terminal.

[1175] Log Management Module

[1176] The server records detailed logs of all actions and events, which can be used for later analysis and troubleshooting.

[1177] For example, the server records detailed data such as which door the user opened at what time, and which lights were turned on at what time, and stores this information for future reference.

[1178] Specific example details

[1179] 1. Nighttime scenario

[1180] If the user wakes up at 2am to go to the bathroom

[1181] The server receives data from the door sensors in real time.

[1182] The server instantly switches on the lights smoothly to allow the user to move safely.

[1183] The server obtains temperature data from the temperature sensor in the room and issues instructions to adjust the temperature to the appropriate level.

[1184] If the door is left open for a long period of time, the server determines that it is an erroneous door opening and closing and instantly sends an alert to the caregiver's device.

[1185] The device notifies the caregiver of the received alert.

[1186] Finally, the server logs every step in detail and stores it for future reference.

[1187] Examples of prompt statements

[1188] Below are some example prompts to input to a generative AI model:

[1189] "Describe a smart home system that automatically turns on the lights, maintains the appropriate room temperature, and notifies caregivers as needed when a user opens the door to go to the bathroom at night."

[1190] In this way, the present invention aims to reduce the burden on caregivers while ensuring the comfort and safety of the user.

[1191] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1192] Specific processing flow of the program

[1193] Sensor Data Collection Module

[1194] Step 1:

[1195] Users install temperature sensors, light sensors, door sensors, etc. in their living spaces to monitor the user's behavior and environmental conditions in real time.

[1196] Input: Sensor installation area information

[1197] Operation: Sensor device installation and initial setup

[1198] Step 2:

[1199] The server collects real-time data from each sensor, including the room temperature, lighting status, and door open / close status.

[1200] Input: Real-time data from each sensor

[1201] What it does: polls and collects data

[1202] Output: Real-time sensor data

[1203] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1204] AI Learning Module

[1205] Step 3:

[1206] The server runs a learning module using artificial intelligence (AI) based on the collected sensor data. The AI ​​module learns the user's behavioral patterns and makes it possible to predict the user's behavior at specific times and in specific situations.

[1207] Input: Real-time sensor data

[1208] What it does: Analyze data and run learning algorithms

[1209] Output: Learned behavioral patterns

[1210] Example: AI learns the pattern of a user moving to the bedroom around 10 p.m. each night and adjusts the room temperature appropriately at that time.

[1211] Action Generation Module

[1212] Step 4:

[1213] The server generates optimal actions for users based on behavioral patterns obtained from the AI ​​learning module, such as automatically turning on lights or adjusting the temperature.

[1214] Input: Learned behavioral patterns

[1215] Behavior: Condition determination and action generation

[1216] Output: The generated actions

[1217] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[1218] Notification Module

[1219] Step 5:

[1220] The server sends alerts and notifications to the caregiver's device when certain events occur, allowing the caregiver to understand the user's current condition in real time and respond quickly to emergencies.

[1221] Input: A specific event (e.g., door opening and closing at midnight)

[1222] Action: Detecting an event and sending a notification

[1223] Output: Notification sent to caregiver's device

[1224] Example: If a user accidentally opens an external door late at night, the server detects this information and promptly sends a notification to the caregiver's device.

[1225] Log Management Module

[1226] Step 6:

[1227] The server records detailed logs of all actions and events, which are stored for later analysis and troubleshooting.

[1228] Input: Generated actions and detected events

[1229] Operation: Recording and storing data

[1230] Output: Detailed log data

[1231] Example: A server records detailed data such as which door a user opens and at what time, and which lights are turned on and at what time, and stores this information for future reference.

[1232] Thus, each processing step of this program aims to reduce the burden on caregivers while ensuring the comfort and safety of the user.

[1233] (Application example 1)

[1234] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1235] Current physical store management systems lack the ability to automatically adjust environmental factors such as temperature and lighting according to customer visits, resulting in an inconsistent customer experience and a heavy burden on managers to make manual adjustments. Notification functions for abnormalities are also limited, making it difficult to respond quickly. This hinders efficient store operations and improved customer satisfaction.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1237] In this invention, the server includes means for collecting sensor data, means for learning user behavior patterns based on the collected sensor data using artificial intelligence, means for generating actions appropriate for the user based on the learned behavior patterns, means for controlling devices to execute the generated actions, means for notifying an administrator when a specific event occurs, means for recording all actions and events as a log, and means for automatically adjusting environmental factors (temperature, lighting, etc.) in the physical store according to customer behavior patterns, thereby enabling effective environmental management and rapid response to abnormalities.

[1238] "Sensor data" refers to environmental information collected by sensors, such as temperature, lighting, human presence, and door opening and closing status.

[1239] "Artificial intelligence" is the technology that enables computer systems to analyze data patterns, learn, and make predictions.

[1240] A "behavioral pattern" is a collection of data that indicates the tendency of regular behavior of a user or customer.

[1241] "Action" refers to optimal behavior or adjustments generated based on learned behavioral patterns.

[1242] "Device" is a general term for equipment or facilities that are controlled to execute the generated action.

[1243] "Administrator" refers to the person in charge of running the system or store, and who will be notified of events.

[1244] A "log" is a data record that records all actions and events and is used for later analysis and reference.

[1245] The system of the present invention is a smart environment management system that aims to improve the customer experience in physical stores and streamline store management. This system is composed of the following main modules.

[1246] 1. Sensor data collection module

[1247] The server collects data in real time from temperature sensors, lighting sensors, motion sensors, door sensors, etc. installed in the physical store, thereby obtaining environmental information within the store (temperature, lighting, human movement, door opening / closing status).

[1248] 2. AI Learning Module

[1249] The server uses artificial intelligence to learn customer behavior patterns based on collected sensor data. This module uses data analysis and pattern recognition to predict customer movements within the store at specific times.

[1250] 3. Action Generation Module

[1251] The server generates optimal actions based on learned behavioral patterns, such as automatically brightening the lights and maintaining an appropriate temperature when the store is crowded.

[1252] 4. Device Control Module

[1253] The server controls devices (such as lighting equipment, air conditioning equipment, and security devices) to execute the generated actions, thereby providing a comfortable environment for the customer.

[1254] 5. Notification Module

[1255] The server sends an alert to the administrator's device when a specific event occurs. For example, if it detects abnormal activity, it will immediately notify the administrator.

[1256] 6. Log Management Module

[1257] The server records all actions and events in detail and makes them available for later analysis and troubleshooting.

[1258] Hardware and software used

[1259] Hardware: Temperature sensors, lighting sensors, motion sensors, door sensors, lighting equipment, air conditioning equipment, security devices

[1260] Software: Python, data collection API, notification API, log storage function

[1261] Specific examples

[1262] For example, consider the process when a particular area in a store is crowded.

[1263] The server receives congestion data from the motion sensors in real time.

[1264] The server uses an AI learning module to analyze congestion patterns.

[1265] The server uses an action generation module to generate actions to brighten the lights and lower the temperature.

[1266] The server controls lighting fixtures and air conditioning equipment through device control modules.

[1267] As a result of this series of actions, customers can visit the store in a more comfortable environment.

[1268] Prompt Sentence Examples

[1269] For example, by inputting the following prompt sentence into the generative AI model, it is possible to generate a document that improves the action generation algorithm.

[1270] "Please provide a detailed description of appropriate environmental adjustment methods when stores are crowded, based on data obtained from motion sensors and temperature sensors in the physical store."

[1271] Based on this prompt, the generative AI can suggest specific actions and adjustments.

[1272] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1273] Step 1:

[1274] The server collects data in real time from temperature sensors, lighting sensors, motion sensors, door sensors, and other sensors installed in the physical store. Environmental information from each sensor (temperature, lighting, human movement, door opening / closing status) is taken in as input, and sensor data is obtained as output. This sensor data makes it possible to grasp the current situation inside the store.

[1275] Step 2:

[1276] The server uses artificial intelligence (AI) to learn customer behavior patterns based on the collected sensor data. The sensor data is passed as input to an AI learning module, which performs data analysis and pattern recognition. The output includes customer behavior patterns and trends, making it possible to predict customer behavior at specific times and in specific situations.

[1277] Step 3:

[1278] The server generates optimal actions based on the learned behavioral patterns. It uses the behavioral patterns obtained from the AI ​​learning module and real-time sensor data as input. It performs data calculations (adjusting temperature, lighting, etc.) and obtains specific actions as output. These actions include automatically adjusting the temperature and lighting in the store.

[1279] Step 4:

[1280] The server controls devices (such as lighting equipment, air conditioning equipment, and security devices) to execute the generated actions. It receives action instructions from the action generation module as input and sends specific signals to control each device as output. This operation makes the in-store environment more comfortable for customers.

[1281] Step 5:

[1282] The server sends an alert to the administrator's terminal when a specific event occurs. It monitors sensor data and system status as input, and generates notification data as output when an abnormality or specific condition is detected, which is sent to the administrator's terminal. This allows the administrator to quickly respond to the abnormality.

[1283] Step 6:

[1284] The server records all actions and events in detail and saves them as log data. It collects sensor data, action data, notification data, etc. as input and records them in a log file as output. This log data can be used later for analysis and troubleshooting.

[1285] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1286] The system of the present invention combines a smart home solution for dementia patients with an emotion engine that recognizes the user's emotions. The system automatically adjusts various aspects of daily life, such as lighting, temperature, and security systems, to provide a comfortable and safe living environment based on the user's emotional state.

[1287] Program Overview

[1288] The system consists of the following main modules:

[1289] 1. Sensor data collection module

[1290] 2. AI Learning Module

[1291] 3. Action Generation Module

[1292] 4. Emotion Recognition Module

[1293] 5. Notification Module

[1294] 6. Log Management Module

[1295] Explaining program processing in natural language

[1296] Sensor Data Collection Module

[1297] The server periodically collects data from temperature sensors, light sensors, door sensors, etc. This data is used to monitor the user's environment and for subsequent processing.

[1298] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1299] AI Learning Module

[1300] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[1301] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[1302] Action Generation Module

[1303] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[1304] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[1305] Emotion Recognition Module

[1306] The server uses an emotion engine that analyzes the user's facial expressions, tone of voice, and biometric data to recognize the user's emotions. Recognized emotions include stress, anxiety, joy, etc.

[1307] Example: If a user repeatedly engages in stressful behavior, the server recognizes the pattern through an emotion engine and generates corresponding actions.

[1308] Notification Module

[1309] The server sends alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[1310] Example: If the emotion engine recognizes that the user is in a stressful state, the server sends a notification to the terminal to notify the caregiver.

[1311] Log Management Module

[1312] The server logs all actions, sensor data, and emotion recognition results, making them available for later analysis and troubleshooting.

[1313] Example: A server logs details such as which door a user opens at what time, which lights are turned on at what time, and how the user's emotional state changes, and stores this information for future reference.

[1314] Specific example details

[1315] 1. Nighttime scenario

[1316] Consider a situation where a user wakes up at 2am to go to the bathroom.

[1317] Step 1:

[1318] The server receives door opening / closing data from the door sensor at 2 a.m.

[1319] Step 2:

[1320] The server instantly switches on the lights smoothly, allowing the user to move safely.

[1321] Step 3:

[1322] The server obtains temperature data from the temperature sensor in the room and sends instructions to adjust the temperature to the appropriate level.

[1323] Step 4:

[1324] The server uses an emotion recognition module to check whether the user's emotions change during the process of going to the toilet.

[1325] Step 5:

[1326] If the user is feeling stressed or anxious, the server notifies the caregiver's terminal of that information.

[1327] Step 6:

[1328] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[1329] Step 7:

[1330] The server logs every step in detail and stores it for later analysis.

[1331] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[1332] The processing flow will be explained below.

[1333] Program processing flow

[1334] Sensor Data Collection Module

[1335] Step 1:

[1336] The server periodically requests data from temperature sensors, light sensors, and door sensors.

[1337] Step 2:

[1338] The server receives the real-time data sent from each sensor.

[1339] Step 3:

[1340] The server stores the received data in a database.

[1341] AI Learning Module

[1342] Step 1:

[1343] The server retrieves historical sensor data from the database and performs preprocessing, which includes imputing missing values ​​and normalizing the data.

[1344] Step 2:

[1345] The server runs machine learning algorithms on the prepared data to learn user behavior patterns.

[1346] Step 3:

[1347] The server updates the obtained model and accumulates behavioral patterns.

[1348] Action Generation Module

[1349] Step 1:

[1350] The server assesses the user's current state based on the sensor data collected in real time, for example, determining whether the user is asleep or awake.

[1351] Step 2:

[1352] The server generates appropriate actions (e.g., adjusting the lights or changing the temperature) based on the learned behavioral patterns.

[1353] Step 3:

[1354] The server sends instructions to the corresponding devices to execute the generated actions.

[1355] Emotion Recognition Module

[1356] Step 1:

[1357] The server collects the user's facial expressions, tone of voice, and biometric data (e.g., heart rate and skin potential) in real time.

[1358] Step 2:

[1359] The server analyzes the collected data through an emotion engine to recognize the user's emotion.

[1360] Step 3:

[1361] The server modifies or corrects system actions and notifications based on the recognized emotions.

[1362] Notification Module

[1363] Step 1:

[1364] The server monitors whether a specific event has occurred (for example, a suspicious door opening or the user's state of stress).

[1365] Step 2:

[1366] The server generates an alert when an event that meets the condition occurs.

[1367] Step 3:

[1368] The server transmits the generated alert to the caregiver's terminal.

[1369] Step 4:

[1370] The device notifies the caregiver of the received alert so that they can understand the situation.

[1371] Log Management Module

[1372] Step 1:

[1373] The server generates logs of all actions, events, and emotion recognition results.

[1374] Step 2:

[1375] The server stores the generated logs in a database with a timestamp.

[1376] Step 3:

[1377] The server periodically analyzes logs to monitor system performance, detect abnormalities, and identify areas for improvement.

[1378] Specific example details

[1379] 1. Nighttime scenario

[1380] The user wakes up at 2am to go to the bathroom.

[1381] Step 1:

[1382] The server receives door opening / closing data from the door sensor at 2 a.m.

[1383] Step 2:

[1384] The server stores the data in a database and begins processing it in real time.

[1385] Step 3:

[1386] The server recognizes that the user is moving from the bedroom to the bathroom and immediately sends a command to the lighting device to turn on the lights.

[1387] Step 4:

[1388] At the same time, the server obtains temperature data from the room's temperature sensor and sends instructions to the heating or cooling device to adjust the temperature to the appropriate level.

[1389] Step 5:

[1390] The server collects the user's facial expressions, tone of voice, and biometric data through an emotion recognition module to recognize the user's emotions.

[1391] Step 6:

[1392] If the user feels stress or anxiety while going to the toilet, the server notifies the caregiver's terminal of that information in real time.

[1393] Step 7:

[1394] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[1395] Step 8:

[1396] The server logs every step in detail and stores it for later analysis.

[1397] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[1398] Example 2

[1399] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1400] In modern times, smart home systems for dementia patients are important for supporting their daily lives. However, conventional smart home systems have limited automation capabilities based on the user's behavioral patterns and lack the ability to respond to the user's emotional changes. Therefore, there is a need for a system that can monitor the emotional state of dementia patients in real time and adjust the environment accordingly.

[1401] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1402] In this invention, the server includes means for collecting sensor data, means for learning user behavior patterns based on the collected sensor data using artificial intelligence, and means for generating actions suitable for the user based on the learned behavior patterns, thereby making it possible to monitor the user's behavior and emotional state in real time and optimize the environment based on the results.

[1403] "Sensor data" is a general term for various environmental data obtained through sensors, such as temperature, lighting, and door opening / closing status.

[1404] "Artificial intelligence" is a general term for algorithms and technologies that analyze sensor data and learn and predict user behavioral patterns and emotions.

[1405] "Behavioral patterns" refer to the tendencies and habits of a series of actions in a user's daily life.

[1406] "Action" refers to an instruction such as adjusting lighting or setting temperature that the system executes based on the user's behavioral patterns and emotion recognition results.

[1407] "Device" refers to equipment or facilities that can be controlled by the system, such as lights, thermostats, and door locks.

[1408] "Emotion" refers to the user's psychological state, which is recognized based on facial expressions, tone of voice, and biometric data.

[1409] "Recognition" refers to the process of using sensor data and artificial intelligence to identify and understand user behavior and emotions.

[1410] "Notification" refers to an alert or message sent to a caregiver or relevant third party when a specified condition is met.

[1411] "Log" refers to a record of all actions and events performed by the system, which can be used for later analysis and troubleshooting.

[1412] The present invention is a smart home system targeted at dementia patients, which is capable of recognizing the user's behavioral patterns and emotions and automatically adjusting the living environment based on them.

[1413] Explaining program processing in natural language

[1414] The system is implemented using the following hardware and software.

[1415] Hardware: Temperature sensor, light sensor, door sensor, camera, microphone

[1416] Software: Server (database, AI learning module, emotion recognition engine, notification module, log management module), terminal (smartphone, tablet)

[1417] Sensor Data Collection Module

[1418] The server periodically collects data from temperature sensors, light sensors, door sensors, cameras, and microphones to monitor the user's environment and use it for subsequent processing.

[1419] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1420] AI Learning Module

[1421] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[1422] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[1423] Action Generation Module

[1424] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[1425] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[1426] Emotion Recognition Module

[1427] The server uses an emotion engine that analyzes the user's facial expressions, tone of voice, and biometric data to recognize the user's emotions. Recognized emotions include stress, anxiety, joy, etc.

[1428] Example: If a user repeatedly engages in stressful behavior, the server recognizes the pattern through an emotion engine and generates corresponding actions.

[1429] Notification Module

[1430] The server sends alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[1431] Example: If the emotion engine recognizes that the user is in a stressful state, the server sends a notification to the terminal to notify the caregiver.

[1432] Log Management Module

[1433] The server logs all actions, sensor data, and emotion recognition results, making them available for later analysis and troubleshooting.

[1434] Example: A server logs details such as which door a user opens at what time, which lights are turned on at what time, and how the user's emotional state changes, and stores this information for future reference.

[1435] Specific example details

[1436] 1. Nighttime scenario

[1437] Consider a situation where a user wakes up at 2am to go to the bathroom.

[1438] Step 1:

[1439] The server receives door opening / closing data from the door sensor at 2 a.m.

[1440] Step 2:

[1441] The server instantly switches on the lights smoothly, allowing the user to move safely.

[1442] Step 3:

[1443] The server obtains temperature data from the temperature sensor in the room and sends instructions to adjust the temperature to the appropriate level.

[1444] Step 4:

[1445] The server uses an emotion recognition module to check whether the user's emotions change during the process of going to the toilet.

[1446] Step 5:

[1447] If the user is feeling stressed or anxious, the server notifies the caregiver's terminal of that information.

[1448] Step 6:

[1449] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[1450] Step 7:

[1451] The server logs every step in detail and stores it for later analysis.

[1452] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[1453] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1454] Step 1:

[1455] The server periodically collects data from temperature sensors, light sensors, door sensors, cameras, and microphones.

[1456] Input: Real-time data from various sensors

[1457] Data processing: Taking the raw data sent from the sensors and storing it in a database

[1458] Output: The database contains the latest environmental information and user movements.

[1459] Specific operation: Every second, the server collects data on temperature, lighting status, door opening / closing status, and user image and voice data.

[1460] Step 2:

[1461] The server pre-processes the collected sensor data.

[1462] Input: Sensor data stored in a database

[1463] Data processing: removing outliers, normalizing data, and aggregating data by time

[1464] Output: Outlier-corrected and standardized data

[1465] Specific operation: The server filters out outliers from the acquired data and normalizes it. For example, if a temperature sensor shows an extreme value, it will exclude that data.

[1466] Step 3:

[1467] The server runs an AI learning module using the preprocessed data.

[1468] Input: Preprocessed sensor data

[1469] Data calculation: Input data into machine learning models to analyze user behavior patterns

[1470] Output: Learned behavioral patterns

[1471] How it works: The server uses deep learning algorithms to learn user behavior patterns based on data from the past week. For example, it uses TensorFlow or PyTorch.

[1472] Step 4:

[1473] The server uses the learned model to predict user behavior.

[1474] Input: trained model, current sensor data

[1475] Data arithmetic: Predicting the most likely actions based on current data

[1476] Output: Predicted behavior pattern

[1477] Specific operation: The server inputs current data into the trained model and predicts, for example, that "the user will move to the bedroom at 10 p.m."

[1478] Step 5:

[1479] The server recognizes emotions by analyzing the user's facial expressions, tone of voice, and biometric data.

[1480] Input: Data from camera, microphone, and biometric sensors

[1481] Data Computing: Sentiment Analysis with Emotion Recognition Algorithms

[1482] Output: Perceived emotional state (e.g., stress, anxiety, joy)

[1483] Specific operation: The server uses image processing and voice analysis technology (e.g., Emotion AI) to analyze the user's emotions in real time.

[1484] Step 6:

[1485] The server generates appropriate actions based on the predicted behavior and the recognized emotions.

[1486] Input: predicted behavioral patterns, emotion recognition results

[1487] Data calculations: Determine optimal actions based on behavioral patterns and emotional states

[1488] Output: Specific action instructions (e.g., adjust lighting, set temperature, play music)

[1489] Specific operation: If the user moves to the bedroom feeling stressed, the server issues an instruction to play soft lighting and relaxing music.

[1490] Step 7:

[1491] The server sends notifications to the caregiver's terminal when certain conditions are met.

[1492] Input: Emotion recognition results, occurrence of specific events

[1493] Data calculation: Check notification conditions

[1494] Output: Notification message to caregiver device

[1495] Specific operation: When a user is judged to be under "high stress," the server immediately sends a push notification to the caregiver's smartphone to inform them of the situation.

[1496] Step 8:

[1497] The server records and stores all actions, sensor data, and emotion recognition results as logs.

[1498] Input: Output data for each step

[1499] Data processing: Record data with time stamps

[1500] Output: Detailed log data

[1501] Specific operation: The server records various data (e.g., the room temperature is 24 degrees at 10 p.m., the lights are on, the user's stress level is 50%, etc.) and stores it in secure data storage.

[1502] (Application example 2)

[1503] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1504] While existing smart home systems focus on automating the behavioral patterns and environmental settings of dementia patients, they lack the ability to respond quickly and appropriately to emotional changes. Furthermore, in brick-and-mortar stores, it is difficult for employees to instantly grasp a customer's emotional state and respond appropriately, creating challenges in improving customer satisfaction. The present invention aims to solve these challenges by providing a smart system that provides the optimal environment and customer service according to the user's emotional state.

[1505] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting sensor data, means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, means for generating an action appropriate for the user based on the learned behavioral patterns, means for controlling a device to execute the generated action, means for notifying a caregiver when a specific event occurs, means for recording all actions and events as a log, means for recognizing the user's emotions, means for adjusting actions based on the user's emotional state, and means for sending a notification regarding the user's emotional state to an employee in a physical store. This enables optimal environment settings and customer service responses according to the user's emotional state.

[1506] "Sensor data" is data used to measure environmental information such as temperature, lighting, and door opening and closing status.

[1507] "Artificial intelligence" is a technology that learns and analyzes collected data to predict user behavior patterns.

[1508] "Behavior patterns" are data that indicate the tendency of a user to repeat certain time periods or actions in their daily lives.

[1509] An "action" is an instruction for controlling environment settings or devices based on the user's behavioral patterns or emotional state.

[1510] "Devices" are appliances used in homes and brick-and-mortar stores, such as lighting, heating, cooling, and security systems.

[1511] "Emotion recognition" is the process of determining a user's emotional state by analyzing their facial expressions, tone of voice, and biometric data.

[1512] A "specific event" is an important occurrence that should be dealt with by a caregiver or employee, such as abnormal behavior or emotional changes of the user.

[1513] "Notification" refers to sending alerts or messages to caregivers or employees when certain events occur.

[1514] A "log" is data that records all actions performed by the system, sensor data, and emotion recognition results.

[1515] "Employees" are staff who deal with customers in physical stores.

[1516] A "physical store" is a facility that has a physical store space and provides services and products to customers.

[1517] A "smart display" is a device that displays information in real time, allowing employees to take appropriate action.

[1518] This invention is a smart system that provides optimal environment settings and appropriate customer service based on the user's sensor data and emotional state. This system mainly consists of a sensor data collection module, an AI learning module, an action generation module, an emotion recognition module, a notification module, and a log management module.

[1519] The server periodically collects data from temperature sensors, lighting sensors, door sensors, etc., and uses this data to monitor the user's environment. The collected sensor data is analyzed by an AI learning module to learn the user's behavioral patterns and preferences. This allows the system to generate actions, such as automatically turning on appropriate lights when the user goes to the bathroom at night.

[1520] The server also uses an emotion recognition module to analyze the user's facial expressions, tone of voice, and biometric data to recognize their emotional state. Based on the recognized emotion, the server can provide an environment that helps the user relax, or in the case of a physical store, provide appropriate customer service advice to employees.

[1521] For example, if a user in a physical store has a dissatisfied expression on their face, the emotion recognition module will detect this and the server will send a notification to the employee's smart display saying, "The customer looks dissatisfied. They may need help." The employee who receives this notification can respond to the customer promptly and appropriately, improving customer satisfaction.

[1522] The server logs the various actions and events generated for future analysis and troubleshooting, allowing for continuous improvement of the system's accuracy and efficiency.

[1523] An example of a prompt sentence is, "A customer who is currently in a low mood seems to be having trouble with the product section. What is the recommended way to respond?", which is sent to the generative AI model. Based on this prompt, the AI ​​model generates advice such as "Ask the customer kindly" or "Provide additional explanation" and provides it to employees.

[1524] In this way, the present invention provides an advanced smart system that recognizes the user's emotional state in real time and then performs appropriate environmental settings and customer responses based on that recognition.

[1525] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1526] Step 1:

[1527] The server periodically collects various sensor data, such as temperature, lighting, and door sensors. This data is acquired by the sensor data collection module. The input is temperature, lighting, and door open / close status data, and the output is a collection of sensor data.

[1528] Step 2:

[1529] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences. The input is a collection of sensor data, and the output is the learning results that show the user's behavioral patterns. This makes it possible to predict what actions the user will take at a certain time.

[1530] Step 3:

[1531] The server uses an action generation module to generate optimal actions for the user based on the learned behavioral patterns. The input is the learned behavioral patterns, and the output is specific instructions or actions for the user. These actions include turning on the lights or adjusting the temperature.

[1532] Step 4:

[1533] The server uses an emotion recognition module to analyze the user's facial expressions, tone of voice, and biometric data to recognize emotions. The input is camera footage and audio data, and the output is data indicating the user's emotional state. Specifically, the camera footage is processed with a facial recognition algorithm to classify emotions.

[1534] Step 5:

[1535] The server adjusts the actions generated by the action generation module based on the recognized emotional state. The input is the emotion recognition data and the generated action, and the output is the action adjusted according to the emotion. This allows the environment to be configured according to the user's emotional state.

[1536] Step 6:

[1537] In a physical store, the server sends notifications about the emotional state to the employee's device. The input is the coordinated action and emotion recognition data, and the output is the notification content. Specifically, the employee's smart display displays a message saying, "The customer appears unhappy and may need help."

[1538] Step 7:

[1539] The server logs all actions and events and stores them for future analysis and troubleshooting. The input is various action and event data, and the output is a log file. This allows for continuous improvement and reliability of the system.

[1540] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1541] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1542] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1543] [Fourth embodiment]

[1544] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1545] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1546] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1547] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1548] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1549] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1550] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1551] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1552] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1553] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1554] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1555] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1556] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1557] The system of the present invention is a smart home solution for dementia patients that automatically adjusts various aspects of daily life, such as lighting, temperature, and security systems, to provide a comfortable and safe living environment.

[1558] Program Overview

[1559] The system consists of the following main modules:

[1560] 1. Sensor data collection module

[1561] 2. AI Learning Module

[1562] 3. Action Generation Module

[1563] 4. Notification Module

[1564] 5. Log Management Module

[1565] Explaining program processing in natural language

[1566] Sensor Data Collection Module

[1567] The server collects real-time data from various sensors (temperature sensors, lighting sensors, door sensors, etc.) installed in the user's environment, including temperature, lighting status, and door open / close status.

[1568] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1569] AI Learning Module

[1570] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[1571] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[1572] Action Generation Module

[1573] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[1574] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[1575] Notification Module

[1576] The server can send alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[1577] Example: If a user accidentally opens an external door in the middle of the night, the server immediately sends a notification to the device to notify the caregiver.

[1578] Log Management Module

[1579] The server logs all actions and sensor data, making it available for later analysis and troubleshooting.

[1580] Example: A server records detailed data such as which door a user opens and at what time, and which lights are turned on and at what time, and stores this information for future reference.

[1581] Specific example details

[1582] 1. Nighttime scenario

[1583] Consider a situation where a user wakes up at 2am to go to the bathroom.

[1584] The server receives data from the door sensors in real time.

[1585] The server immediately switches on the lights smoothly, allowing the user to move safely.

[1586] The server also obtains temperature data from the room's temperature sensor and issues instructions to adjust the temperature to the appropriate level.

[1587] If the door remains open for a long period of time, the server determines that it is an erroneous door opening and closing and instantly sends an alert to the terminal.

[1588] The terminal notifies the caregiver of the received alert.

[1589] Finally, the server logs every step in detail and stores it for future reference.

[1590] In this way, the system reduces the burden on caregivers while ensuring the comfort and safety of the user.

[1591] The processing flow will be explained below.

[1592] Program processing flow

[1593] Sensor Data Collection Module

[1594] Step 1:

[1595] The server periodically requests data from temperature sensors, light sensors, door sensors, etc.

[1596] Step 2:

[1597] The server receives the real-time data sent from each sensor.

[1598] Step 3:

[1599] The server stores the received data in a database.

[1600] AI Learning Module

[1601] Step 1:

[1602] The server retrieves historical sensor data from the database and performs preprocessing, which includes imputing missing values ​​and normalizing the data.

[1603] Step 2:

[1604] The server runs machine learning algorithms on the prepared data to learn user behavior patterns.

[1605] Step 3:

[1606] The server updates the obtained model and accumulates behavioral patterns.

[1607] Action Generation Module

[1608] Step 1:

[1609] The server assesses the user's current state based on the sensor data collected in real time, for example, determining whether the user is asleep or awake.

[1610] Step 2:

[1611] The server generates appropriate actions (e.g., adjusting the lights or changing the temperature) based on the learned behavioral patterns.

[1612] Step 3:

[1613] The server sends instructions to the corresponding devices to execute the generated actions.

[1614] Notification Module

[1615] Step 1:

[1616] The server monitors whether a particular event (e.g., a suspicious door opening or closing) has occurred.

[1617] Step 2:

[1618] The server generates an alert when an event occurs that meets the set conditions.

[1619] Step 3:

[1620] The server transmits the generated alert to the caregiver's terminal.

[1621] Step 4:

[1622] The device notifies the caregiver of the received alert so that they can understand the situation.

[1623] Log Management Module

[1624] Step 1:

[1625] The server generates a log of all actions and events.

[1626] Step 2:

[1627] The server stores the generated logs in a database with a timestamp.

[1628] Step 3:

[1629] The server periodically analyzes logs to monitor system performance, detect abnormalities, and identify areas for improvement.

[1630] Specific example details

[1631] For nighttime scenarios:

[1632] 1. The user wakes up at 2am to go to the bathroom.

[1633] Step 1:

[1634] The server receives door opening / closing data from the door sensor at 2 a.m.

[1635] Step 2:

[1636] The server stores the data in a database and begins processing it in real time.

[1637] Step 3:

[1638] The server recognizes that the user is moving from the bedroom to the bathroom and immediately sends a command to the lighting device to turn on the lights.

[1639] Step 4:

[1640] The server simultaneously obtains temperature data from the room's temperature sensor and sends instructions to the heating or cooling device to adjust the temperature to the desired level.

[1641] Step 5:

[1642] The server determines that the door has been opened or closed incorrectly and quickly sends an alert to the terminal.

[1643] Step 6:

[1644] The device notifies the caregiver of the received alert so that they can understand the situation.

[1645] Step 7:

[1646] The server logs every step in detail and stores it for future analysis.

[1647] This system ensures the comfort and safety of the user while reducing the burden on caregivers.

[1648] Example 1

[1649] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1650] Maintaining a comfortable and safe living environment for dementia patients places a heavy burden on caregivers. There is a need for an efficient system that can predict users' behavioral patterns and automatically generate and execute appropriate actions. The goal of this system is to reduce the burden on caregivers while ensuring the safety and comfort of users.

[1651] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1652] In this invention, the server includes means for collecting data from multiple sensors installed in the environment, means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, means for generating actions appropriate for the user based on the learned behavioral patterns, means for controlling devices to execute the generated actions, means for notifying a caregiver when a specific event occurs, and means for recording all actions and events as a log. This makes it possible to predict the user's behavior and take appropriate measures to ensure the user's safety and comfort and reduce the burden on the caregiver.

[1653] "Sensor data" is a general term for various data such as temperature, lighting, and door opening / closing status obtained from sensors installed in the environment.

[1654] "Artificial intelligence" is a technology that allows computers to imitate human intelligence, learn, reason, and assist in problem-solving.

[1655] A "behavior pattern" is a collection of data that indicates the tendencies and regularities of the user's daily behavior.

[1656] An "action" is a response or operation that the system automatically generates and executes based on the user's behavioral patterns.

[1657] A "device" is a hardware device controlled by the system, and specifically includes lighting devices, temperature control devices, door lock devices, etc.

[1658] An "event" refers to a specific action or change in state of the user, such as opening and closing a door, a sudden change in temperature, or movement during the night.

[1659] "Notifications" are alerts or messages sent to caregivers when certain events occur.

[1660] "Log" means detailed historical data of all actions and events recorded by the system for later analysis and troubleshooting.

[1661] A "caregiver" is a person whose role is to support the lives of people with dementia and provide the necessary care.

[1662] "Environment" refers to the residence where the user lives and the surrounding area, and refers to the space where the sensor is installed.

[1663] This invention relates to a smart home solution for dementia patients. The system automatically adjusts various aspects of daily life such as lighting, temperature, and security systems to provide a comfortable and safe living environment.

[1664] Program Overview

[1665] The system includes the following main modules:

[1666] 1. Sensor data collection module

[1667] 2. AI Learning Module

[1668] 3. Action Generation Module

[1669] 4. Notification Module

[1670] 5. Log Management Module

[1671] Sensor Data Collection Module

[1672] Users install temperature sensors, lighting sensors, door sensors, etc. in their living spaces. These sensors collect the user's environmental data (temperature, lighting status, door open / close status, etc.) in real time. This sensor data is sent to a server and recorded.

[1673] As a specific example, when a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1674] AI Learning Module

[1675] The server uses collected sensor data to run a learning module using artificial intelligence (AI) to learn the user's behavioral patterns and environmental preferences, making it possible to predict the user's behavior at specific times and in specific situations.

[1676] As a specific example, a server that learns that a user moves to their bedroom around 10 p.m. every night will set the room temperature to be adjusted appropriately during that time.

[1677] Action Generation Module

[1678] The server generates appropriate actions based on the behavioral patterns obtained from the AI ​​learning module, such as automatically turning on the lights when the user wakes up.

[1679] As a specific example, the server learns that the user wakes up at 7:00 every morning and sets the lights to gradually brighten accordingly.

[1680] Notification Module

[1681] The server sends alerts and notifications to the caregiver's device when certain events occur, allowing the caregiver to understand the user's real-time situation and respond quickly.

[1682] As a specific example, if a user accidentally opens an external door in the middle of the night, the server quickly detects this information and sends a notification to the caregiver's terminal.

[1683] Log Management Module

[1684] The server records detailed logs of all actions and events, which can be used for later analysis and troubleshooting.

[1685] For example, the server records detailed data such as which door the user opened at what time, and which lights were turned on at what time, and stores this information for future reference.

[1686] Specific example details

[1687] 1. Nighttime scenario

[1688] If the user wakes up at 2am to go to the bathroom

[1689] The server receives data from the door sensors in real time.

[1690] The server instantly switches on the lights smoothly to allow the user to move safely.

[1691] The server obtains temperature data from the temperature sensor in the room and issues instructions to adjust the temperature to the appropriate level.

[1692] If the door is left open for a long period of time, the server determines that it is an erroneous door opening and closing and instantly sends an alert to the caregiver's device.

[1693] The device notifies the caregiver of the received alert.

[1694] Finally, the server logs every step in detail and stores it for future reference.

[1695] Examples of prompt statements

[1696] Below are some example prompts to input to a generative AI model:

[1697] "Describe a smart home system that automatically turns on the lights, maintains the appropriate room temperature, and notifies caregivers as needed when a user opens the door to go to the bathroom at night."

[1698] In this way, the present invention aims to reduce the burden on caregivers while ensuring the comfort and safety of the user.

[1699] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1700] Specific processing flow of the program

[1701] Sensor Data Collection Module

[1702] Step 1:

[1703] Users install temperature sensors, light sensors, door sensors, etc. in their living spaces to monitor the user's behavior and environmental conditions in real time.

[1704] Input: Sensor installation area information

[1705] Operation: Sensor device installation and initial setup

[1706] Step 2:

[1707] The server collects real-time data from each sensor, including the room temperature, lighting status, and door open / close status.

[1708] Input: Real-time data from each sensor

[1709] What it does: polls and collects data

[1710] Output: Real-time sensor data

[1711] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1712] AI Learning Module

[1713] Step 3:

[1714] The server runs a learning module using artificial intelligence (AI) based on the collected sensor data. The AI ​​module learns the user's behavioral patterns and makes it possible to predict the user's behavior at specific times and in specific situations.

[1715] Input: Real-time sensor data

[1716] What it does: Analyze data and run learning algorithms

[1717] Output: Learned behavioral patterns

[1718] Example: AI learns the pattern of a user moving to the bedroom around 10 p.m. each night and adjusts the room temperature appropriately at that time.

[1719] Action Generation Module

[1720] Step 4:

[1721] The server generates optimal actions for users based on behavioral patterns obtained from the AI ​​learning module, such as automatically turning on lights or adjusting the temperature.

[1722] Input: Learned behavioral patterns

[1723] Behavior: Condition determination and action generation

[1724] Output: The generated actions

[1725] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[1726] Notification Module

[1727] Step 5:

[1728] The server sends alerts and notifications to the caregiver's device when certain events occur, allowing the caregiver to understand the user's current condition in real time and respond quickly to emergencies.

[1729] Input: A specific event (e.g., door opening and closing at midnight)

[1730] Action: Detecting an event and sending a notification

[1731] Output: Notification sent to caregiver's device

[1732] Example: If a user accidentally opens an external door late at night, the server detects this information and promptly sends a notification to the caregiver's device.

[1733] Log Management Module

[1734] Step 6:

[1735] The server records detailed logs of all actions and events, which are stored for later analysis and troubleshooting.

[1736] Input: Generated actions and detected events

[1737] Operation: Recording and storing data

[1738] Output: Detailed log data

[1739] Example: A server records detailed data such as which door a user opens and at what time, and which lights are turned on and at what time, and stores this information for future reference.

[1740] Thus, each processing step of this program aims to reduce the burden on caregivers while ensuring the comfort and safety of the user.

[1741] (Application example 1)

[1742] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1743] Current physical store management systems lack the ability to automatically adjust environmental factors such as temperature and lighting according to customer visits, resulting in an inconsistent customer experience and a heavy burden on managers to make manual adjustments. Notification functions for abnormalities are also limited, making it difficult to respond quickly. This hinders efficient store operations and improved customer satisfaction.

[1744] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1745] In this invention, the server includes means for collecting sensor data, means for learning user behavior patterns based on the collected sensor data using artificial intelligence, means for generating actions appropriate for the user based on the learned behavior patterns, means for controlling devices to execute the generated actions, means for notifying an administrator when a specific event occurs, means for recording all actions and events as a log, and means for automatically adjusting environmental factors (temperature, lighting, etc.) in the physical store according to customer behavior patterns, thereby enabling effective environmental management and rapid response to abnormalities.

[1746] "Sensor data" refers to environmental information collected by sensors, such as temperature, lighting, human presence, and door opening and closing status.

[1747] "Artificial intelligence" is the technology that enables computer systems to analyze data patterns, learn, and make predictions.

[1748] A "behavioral pattern" is a collection of data that indicates the tendency of regular behavior of a user or customer.

[1749] "Action" refers to optimal behavior or adjustments generated based on learned behavioral patterns.

[1750] "Device" is a general term for equipment or facilities that are controlled to execute the generated action.

[1751] "Administrator" refers to the person in charge of running the system or store, and who will be notified of events.

[1752] A "log" is a data record that records all actions and events and is used for later analysis and reference.

[1753] The system of the present invention is a smart environment management system that aims to improve the customer experience in physical stores and streamline store management. This system is composed of the following main modules.

[1754] 1. Sensor data collection module

[1755] The server collects data in real time from temperature sensors, lighting sensors, motion sensors, door sensors, etc. installed in the physical store, thereby obtaining environmental information within the store (temperature, lighting, human movement, door opening / closing status).

[1756] 2. AI Learning Module

[1757] The server uses artificial intelligence to learn customer behavior patterns based on collected sensor data. This module uses data analysis and pattern recognition to predict customer movements within the store at specific times.

[1758] 3. Action Generation Module

[1759] The server generates optimal actions based on learned behavioral patterns, such as automatically brightening the lights and maintaining an appropriate temperature when the store is crowded.

[1760] 4. Device Control Module

[1761] The server controls devices (such as lighting equipment, air conditioning equipment, and security devices) to execute the generated actions, thereby providing a comfortable environment for the customer.

[1762] 5. Notification Module

[1763] The server sends an alert to the administrator's device when a specific event occurs. For example, if it detects abnormal activity, it will immediately notify the administrator.

[1764] 6. Log Management Module

[1765] The server records all actions and events in detail and makes them available for later analysis and troubleshooting.

[1766] Hardware and software used

[1767] Hardware: Temperature sensors, lighting sensors, motion sensors, door sensors, lighting equipment, air conditioning equipment, security devices

[1768] Software: Python, data collection API, notification API, log storage function

[1769] Specific examples

[1770] For example, consider the process when a particular area in a store is crowded.

[1771] The server receives congestion data from the motion sensors in real time.

[1772] The server uses an AI learning module to analyze congestion patterns.

[1773] The server uses an action generation module to generate actions to brighten the lights and lower the temperature.

[1774] The server controls lighting fixtures and air conditioning equipment through device control modules.

[1775] As a result of this series of actions, customers can visit the store in a more comfortable environment.

[1776] Prompt Sentence Examples

[1777] For example, by inputting the following prompt sentence into the generative AI model, it is possible to generate a document that improves the action generation algorithm.

[1778] "Please provide a detailed description of appropriate environmental adjustment methods when stores are crowded, based on data obtained from motion sensors and temperature sensors in the physical store."

[1779] Based on this prompt, the generative AI can suggest specific actions and adjustments.

[1780] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1781] Step 1:

[1782] The server collects data in real time from temperature sensors, lighting sensors, motion sensors, door sensors, and other sensors installed in the physical store. Environmental information from each sensor (temperature, lighting, human movement, door opening / closing status) is taken in as input, and sensor data is obtained as output. This sensor data makes it possible to grasp the current situation inside the store.

[1783] Step 2:

[1784] The server uses artificial intelligence (AI) to learn customer behavior patterns based on the collected sensor data. The sensor data is passed as input to an AI learning module, which performs data analysis and pattern recognition. The output includes customer behavior patterns and trends, making it possible to predict customer behavior at specific times and in specific situations.

[1785] Step 3:

[1786] The server generates optimal actions based on the learned behavioral patterns. It uses the behavioral patterns obtained from the AI ​​learning module and real-time sensor data as input. It performs data calculations (adjusting temperature, lighting, etc.) and obtains specific actions as output. These actions include automatically adjusting the temperature and lighting in the store.

[1787] Step 4:

[1788] The server controls devices (such as lighting equipment, air conditioning equipment, and security devices) to execute the generated actions. It receives action instructions from the action generation module as input and sends specific signals to control each device as output. This operation makes the in-store environment more comfortable for customers.

[1789] Step 5:

[1790] The server sends an alert to the administrator's terminal when a specific event occurs. It monitors sensor data and system status as input, and generates notification data as output when an abnormality or specific condition is detected, which is sent to the administrator's terminal. This allows the administrator to quickly respond to the abnormality.

[1791] Step 6:

[1792] The server records all actions and events in detail and saves them as log data. It collects sensor data, action data, notification data, etc. as input and records them in a log file as output. This log data can be used later for analysis and troubleshooting.

[1793] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1794] The system of the present invention combines a smart home solution for dementia patients with an emotion engine that recognizes the user's emotions. The system automatically adjusts various aspects of daily life, such as lighting, temperature, and security systems, to provide a comfortable and safe living environment based on the user's emotional state.

[1795] Program Overview

[1796] The system consists of the following main modules:

[1797] 1. Sensor data collection module

[1798] 2. AI Learning Module

[1799] 3. Action Generation Module

[1800] 4. Emotion Recognition Module

[1801] 5. Notification Module

[1802] 6. Log Management Module

[1803] Explaining program processing in natural language

[1804] Sensor Data Collection Module

[1805] The server periodically collects data from temperature sensors, light sensors, door sensors, etc. This data is used to monitor the user's environment and for subsequent processing.

[1806] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1807] AI Learning Module

[1808] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[1809] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[1810] Action Generation Module

[1811] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[1812] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[1813] Emotion Recognition Module

[1814] The server uses an emotion engine that analyzes the user's facial expressions, tone of voice, and biometric data to recognize the user's emotions. Recognized emotions include stress, anxiety, joy, etc.

[1815] Example: If a user repeatedly engages in stressful behavior, the server recognizes the pattern through an emotion engine and generates corresponding actions.

[1816] Notification Module

[1817] The server sends alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[1818] Example: If the emotion engine recognizes that the user is in a stressful state, the server sends a notification to the terminal to notify the caregiver.

[1819] Log Management Module

[1820] The server logs all actions, sensor data, and emotion recognition results, making them available for later analysis and troubleshooting.

[1821] Example: A server logs details such as which door a user opens at what time, which lights are turned on at what time, and how the user's emotional state changes, and stores this information for future reference.

[1822] Specific example details

[1823] 1. Nighttime scenario

[1824] Consider a situation where a user wakes up at 2am to go to the bathroom.

[1825] Step 1:

[1826] The server receives door opening / closing data from the door sensor at 2 a.m.

[1827] Step 2:

[1828] The server instantly switches on the lights smoothly, allowing the user to move safely.

[1829] Step 3:

[1830] The server obtains temperature data from the temperature sensor in the room and sends instructions to adjust the temperature to the appropriate level.

[1831] Step 4:

[1832] The server uses an emotion recognition module to check whether the user's emotions change during the process of going to the toilet.

[1833] Step 5:

[1834] If the user is feeling stressed or anxious, the server notifies the caregiver's terminal of that information.

[1835] Step 6:

[1836] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[1837] Step 7:

[1838] The server logs every step in detail and stores it for later analysis.

[1839] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[1840] The processing flow will be explained below.

[1841] Program processing flow

[1842] Sensor Data Collection Module

[1843] Step 1:

[1844] The server periodically requests data from temperature sensors, light sensors, and door sensors.

[1845] Step 2:

[1846] The server receives the real-time data sent from each sensor.

[1847] Step 3:

[1848] The server stores the received data in a database.

[1849] AI Learning Module

[1850] Step 1:

[1851] The server retrieves historical sensor data from the database and performs preprocessing, which includes imputing missing values ​​and normalizing the data.

[1852] Step 2:

[1853] The server runs machine learning algorithms on the prepared data to learn user behavior patterns.

[1854] Step 3:

[1855] The server updates the obtained model and accumulates behavioral patterns.

[1856] Action Generation Module

[1857] Step 1:

[1858] The server assesses the user's current state based on the sensor data collected in real time, for example, determining whether the user is asleep or awake.

[1859] Step 2:

[1860] The server generates appropriate actions (e.g., adjusting the lights or changing the temperature) based on the learned behavioral patterns.

[1861] Step 3:

[1862] The server sends instructions to the corresponding devices to execute the generated actions.

[1863] Emotion Recognition Module

[1864] Step 1:

[1865] The server collects the user's facial expressions, tone of voice, and biometric data (e.g., heart rate and skin potential) in real time.

[1866] Step 2:

[1867] The server analyzes the collected data through an emotion engine to recognize the user's emotion.

[1868] Step 3:

[1869] The server modifies or corrects system actions and notifications based on the recognized emotions.

[1870] Notification Module

[1871] Step 1:

[1872] The server monitors whether a specific event has occurred (for example, a suspicious door opening or the user's state of stress).

[1873] Step 2:

[1874] The server generates an alert when an event that meets the condition occurs.

[1875] Step 3:

[1876] The server transmits the generated alert to the caregiver's terminal.

[1877] Step 4:

[1878] The device notifies the caregiver of the received alert so that they can understand the situation.

[1879] Log Management Module

[1880] Step 1:

[1881] The server generates logs of all actions, events, and emotion recognition results.

[1882] Step 2:

[1883] The server stores the generated logs in a database with a timestamp.

[1884] Step 3:

[1885] The server periodically analyzes logs to monitor system performance, detect abnormalities, and identify areas for improvement.

[1886] Specific example details

[1887] 1. Nighttime scenario

[1888] The user wakes up at 2am to go to the bathroom.

[1889] Step 1:

[1890] The server receives door opening / closing data from the door sensor at 2 a.m.

[1891] Step 2:

[1892] The server stores the data in a database and begins processing it in real time.

[1893] Step 3:

[1894] The server recognizes that the user is moving from the bedroom to the bathroom and immediately sends a command to the lighting device to turn on the lights.

[1895] Step 4:

[1896] At the same time, the server obtains temperature data from the room's temperature sensor and sends instructions to the heating or cooling device to adjust the temperature to the appropriate level.

[1897] Step 5:

[1898] The server collects the user's facial expressions, tone of voice, and biometric data through an emotion recognition module to recognize the user's emotions.

[1899] Step 6:

[1900] If the user feels stress or anxiety while going to the toilet, the server notifies the caregiver's terminal of that information in real time.

[1901] Step 7:

[1902] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[1903] Step 8:

[1904] The server logs every step in detail and stores it for later analysis.

[1905] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[1906] Example 2

[1907] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1908] In modern times, smart home systems for dementia patients are important for supporting their daily lives. However, conventional smart home systems have limited automation capabilities based on the user's behavioral patterns and lack the ability to respond to the user's emotional changes. Therefore, there is a need for a system that can monitor the emotional state of dementia patients in real time and adjust the environment accordingly.

[1909] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1910] In this invention, the server includes means for collecting sensor data, means for learning user behavior patterns based on the collected sensor data using artificial intelligence, and means for generating actions suitable for the user based on the learned behavior patterns, thereby making it possible to monitor the user's behavior and emotional state in real time and optimize the environment based on the results.

[1911] "Sensor data" is a general term for various environmental data obtained through sensors, such as temperature, lighting, and door opening / closing status.

[1912] "Artificial intelligence" is a general term for algorithms and technologies that analyze sensor data and learn and predict user behavioral patterns and emotions.

[1913] "Behavioral patterns" refer to the tendencies and habits of a series of actions in a user's daily life.

[1914] "Action" refers to an instruction such as adjusting lighting or setting temperature that the system executes based on the user's behavioral patterns and emotion recognition results.

[1915] "Device" refers to equipment or facilities that can be controlled by the system, such as lights, thermostats, and door locks.

[1916] "Emotion" refers to the user's psychological state, which is recognized based on facial expressions, tone of voice, and biometric data.

[1917] "Recognition" refers to the process of using sensor data and artificial intelligence to identify and understand user behavior and emotions.

[1918] "Notification" refers to an alert or message sent to a caregiver or relevant third party when a specified condition is met.

[1919] "Log" refers to a record of all actions and events performed by the system, which can be used for later analysis and troubleshooting.

[1920] The present invention is a smart home system targeted at dementia patients, which is capable of recognizing the user's behavioral patterns and emotions and automatically adjusting the living environment based on them.

[1921] Explaining program processing in natural language

[1922] The system is implemented using the following hardware and software.

[1923] Hardware: Temperature sensor, light sensor, door sensor, camera, microphone

[1924] Software: Server (database, AI learning module, emotion recognition engine, notification module, log management module), terminal (smartphone, tablet)

[1925] Sensor Data Collection Module

[1926] The server periodically collects data from temperature sensors, light sensors, door sensors, cameras, and microphones to monitor the user's environment and use it for subsequent processing.

[1927] Example: When a user opens the door to go to the toilet at night, the door sensor detects the opening and closing and sends the data to the server.

[1928] AI Learning Module

[1929] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences, making it possible to predict certain user behavioral trends.

[1930] Example: The server learns the pattern of the user moving to the bedroom around 10 p.m. every night and automatically adjusts the room temperature to a comfortable temperature during that time.

[1931] Action Generation Module

[1932] The server generates optimal actions based on the learned behavioral patterns, such as turning on appropriate lights when the user starts an activity during this time period.

[1933] Example: The server learns that the user wakes up at 7am every morning and sets the lights to gradually brighten accordingly.

[1934] Emotion Recognition Module

[1935] The server uses an emotion engine that analyzes the user's facial expressions, tone of voice, and biometric data to recognize the user's emotions. Recognized emotions include stress, anxiety, joy, etc.

[1936] Example: If a user repeatedly engages in stressful behavior, the server recognizes the pattern through an emotion engine and generates corresponding actions.

[1937] Notification Module

[1938] The server sends alerts and notifications to the caregiver's device when certain conditions are met, allowing the caregiver to understand the user's condition in real time.

[1939] Example: If the emotion engine recognizes that the user is in a stressful state, the server sends a notification to the terminal to notify the caregiver.

[1940] Log Management Module

[1941] The server logs all actions, sensor data, and emotion recognition results, making them available for later analysis and troubleshooting.

[1942] Example: A server logs details such as which door a user opens at what time, which lights are turned on at what time, and how the user's emotional state changes, and stores this information for future reference.

[1943] Specific example details

[1944] 1. Nighttime scenario

[1945] Consider a situation where a user wakes up at 2am to go to the bathroom.

[1946] Step 1:

[1947] The server receives door opening / closing data from the door sensor at 2 a.m.

[1948] Step 2:

[1949] The server instantly switches on the lights smoothly, allowing the user to move safely.

[1950] Step 3:

[1951] The server obtains temperature data from the temperature sensor in the room and sends instructions to adjust the temperature to the appropriate level.

[1952] Step 4:

[1953] The server uses an emotion recognition module to check whether the user's emotions change during the process of going to the toilet.

[1954] Step 5:

[1955] If the user is feeling stressed or anxious, the server notifies the caregiver's terminal of that information.

[1956] Step 6:

[1957] The device then passes the received notification on to the caregiver, allowing them to check the situation.

[1958] Step 7:

[1959] The server logs every step in detail and stores it for later analysis.

[1960] In addition to ensuring the user's safety and comfort, the system is able to monitor and respond to emotional changes, providing an advanced smart home solution that reduces the burden on caregivers overall.

[1961] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1962] Step 1:

[1963] The server periodically collects data from temperature sensors, light sensors, door sensors, cameras, and microphones.

[1964] Input: Real-time data from various sensors

[1965] Data processing: Taking the raw data sent from the sensors and storing it in a database

[1966] Output: The database contains the latest environmental information and user movements.

[1967] Specific operation: Every second, the server collects data on temperature, lighting status, door opening / closing status, and user image and voice data.

[1968] Step 2:

[1969] The server pre-processes the collected sensor data.

[1970] Input: Sensor data stored in a database

[1971] Data processing: removing outliers, normalizing data, and aggregating data by time

[1972] Output: Outlier-corrected and standardized data

[1973] Specific operation: The server filters out outliers from the acquired data and normalizes it. For example, if a temperature sensor shows an extreme value, it will exclude that data.

[1974] Step 3:

[1975] The server runs an AI learning module using the preprocessed data.

[1976] Input: Preprocessed sensor data

[1977] Data calculation: Input data into machine learning models to analyze user behavior patterns

[1978] Output: Learned behavioral patterns

[1979] How it works: The server uses deep learning algorithms to learn user behavior patterns based on data from the past week. For example, it uses TensorFlow or PyTorch.

[1980] Step 4:

[1981] The server uses the learned model to predict user behavior.

[1982] Input: trained model, current sensor data

[1983] Data arithmetic: Predicting the most likely actions based on current data

[1984] Output: Predicted behavior pattern

[1985] Specific operation: The server inputs current data into the trained model and predicts, for example, that "the user will move to the bedroom at 10 p.m."

[1986] Step 5:

[1987] The server recognizes emotions by analyzing the user's facial expressions, tone of voice, and biometric data.

[1988] Input: Data from camera, microphone, and biometric sensors

[1989] Data Computing: Sentiment Analysis with Emotion Recognition Algorithms

[1990] Output: Perceived emotional state (e.g., stress, anxiety, joy)

[1991] Specific operation: The server uses image processing and voice analysis technology (e.g., Emotion AI) to analyze the user's emotions in real time.

[1992] Step 6:

[1993] The server generates appropriate actions based on the predicted behavior and the recognized emotions.

[1994] Input: predicted behavioral patterns, emotion recognition results

[1995] Data calculations: Determine optimal actions based on behavioral patterns and emotional states

[1996] Output: Specific action instructions (e.g., adjust lighting, set temperature, play music)

[1997] Specific operation: If the user moves to the bedroom feeling stressed, the server issues an instruction to play soft lighting and relaxing music.

[1998] Step 7:

[1999] The server sends notifications to the caregiver's terminal when certain conditions are met.

[2000] Input: Emotion recognition results, occurrence of specific events

[2001] Data calculation: Check notification conditions

[2002] Output: Notification message to caregiver device

[2003] Specific operation: When a user is judged to be under "high stress," the server immediately sends a push notification to the caregiver's smartphone to inform them of the situation.

[2004] Step 8:

[2005] The server records and stores all actions, sensor data, and emotion recognition results as logs.

[2006] Input: Output data for each step

[2007] Data processing: Record data with time stamps

[2008] Output: Detailed log data

[2009] Specific operation: The server records various data (e.g., the room temperature is 24 degrees at 10 p.m., the lights are on, the user's stress level is 50%, etc.) and stores it in secure data storage.

[2010] (Application example 2)

[2011] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2012] While existing smart home systems focus on automating the behavioral patterns and environmental settings of dementia patients, they lack the ability to respond quickly and appropriately to emotional changes. Furthermore, in brick-and-mortar stores, it is difficult for employees to instantly grasp a customer's emotional state and respond appropriately, creating challenges in improving customer satisfaction. The present invention aims to solve these challenges by providing a smart system that provides the optimal environment and customer service according to the user's emotional state.

[2013] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting sensor data, means for learning a user's behavioral patterns based on the collected sensor data using artificial intelligence, means for generating an action appropriate for the user based on the learned behavioral patterns, means for controlling a device to execute the generated action, means for notifying a caregiver when a specific event occurs, means for recording all actions and events as a log, means for recognizing the user's emotions, means for adjusting actions based on the user's emotional state, and means for sending a notification regarding the user's emotional state to an employee in a physical store. This enables optimal environment settings and customer service responses according to the user's emotional state.

[2014] "Sensor data" is data used to measure environmental information such as temperature, lighting, and door opening and closing status.

[2015] "Artificial intelligence" is a technology that learns and analyzes collected data to predict user behavior patterns.

[2016] "Behavior patterns" are data that indicate the tendency of a user to repeat certain time periods or actions in their daily lives.

[2017] An "action" is an instruction for controlling environment settings or devices based on the user's behavioral patterns or emotional state.

[2018] "Devices" are appliances used in homes and brick-and-mortar stores, such as lighting, heating, cooling, and security systems.

[2019] "Emotion recognition" is the process of determining a user's emotional state by analyzing their facial expressions, tone of voice, and biometric data.

[2020] A "specific event" is an important occurrence that should be dealt with by a caregiver or employee, such as abnormal behavior or emotional changes of the user.

[2021] "Notification" refers to sending alerts or messages to caregivers or employees when certain events occur.

[2022] A "log" is data that records all actions performed by the system, sensor data, and emotion recognition results.

[2023] "Employees" are staff who deal with customers in physical stores.

[2024] A "physical store" is a facility that has a physical store space and provides services and products to customers.

[2025] A "smart display" is a device that displays information in real time, allowing employees to take appropriate action.

[2026] This invention is a smart system that provides optimal environment settings and appropriate customer service based on the user's sensor data and emotional state. This system mainly consists of a sensor data collection module, an AI learning module, an action generation module, an emotion recognition module, a notification module, and a log management module.

[2027] The server periodically collects data from temperature sensors, lighting sensors, door sensors, etc., and uses this data to monitor the user's environment. The collected sensor data is analyzed by an AI learning module to learn the user's behavioral patterns and preferences. This allows the system to generate actions, such as automatically turning on appropriate lights when the user goes to the bathroom at night.

[2028] The server also uses an emotion recognition module to analyze the user's facial expressions, tone of voice, and biometric data to recognize their emotional state. Based on the recognized emotion, the server can provide an environment that helps the user relax, or in the case of a physical store, provide appropriate customer service advice to employees.

[2029] For example, if a user in a physical store has a dissatisfied expression on their face, the emotion recognition module will detect this and the server will send a notification to the employee's smart display saying, "The customer looks dissatisfied. They may need help." The employee who receives this notification can respond to the customer promptly and appropriately, improving customer satisfaction.

[2030] The server logs the various actions and events generated for future analysis and troubleshooting, allowing for continuous improvement of the system's accuracy and efficiency.

[2031] An example of a prompt sentence is, "A customer who is currently in a low mood seems to be having trouble with the product section. What is the recommended way to respond?", which is sent to the generative AI model. Based on this prompt, the AI ​​model generates advice such as "Ask the customer kindly" or "Provide additional explanation" and provides it to employees.

[2032] In this way, the present invention provides an advanced smart system that recognizes the user's emotional state in real time and then performs appropriate environmental settings and customer responses based on that recognition.

[2033] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2034] Step 1:

[2035] The server periodically collects various sensor data, such as temperature, lighting, and door sensors. This data is acquired by the sensor data collection module. The input is temperature, lighting, and door open / close status data, and the output is a collection of sensor data.

[2036] Step 2:

[2037] The server runs an AI learning module based on the collected sensor data to learn the user's behavioral patterns and preferences. The input is a collection of sensor data, and the output is the learning results that show the user's behavioral patterns. This makes it possible to predict what actions the user will take at a certain time.

[2038] Step 3:

[2039] The server uses an action generation module to generate optimal actions for the user based on the learned behavioral patterns. The input is the learned behavioral patterns, and the output is specific instructions or actions for the user. These actions include turning on the lights or adjusting the temperature.

[2040] Step 4:

[2041] The server uses an emotion recognition module to analyze the user's facial expressions, tone of voice, and biometric data to recognize emotions. The input is camera footage and audio data, and the output is data indicating the user's emotional state. Specifically, the camera footage is processed with a facial recognition algorithm to classify emotions.

[2042] Step 5:

[2043] The server adjusts the actions generated by the action generation module based on the recognized emotional state. The input is the emotion recognition data and the generated action, and the output is the action adjusted according to the emotion. This allows the environment to be configured according to the user's emotional state.

[2044] Step 6:

[2045] In a physical store, the server sends notifications about the emotional state to the employee's device. The input is the coordinated action and emotion recognition data, and the output is the notification content. Specifically, the employee's smart display displays a message saying, "The customer appears unhappy and may need help."

[2046] Step 7:

[2047] The server logs all actions and events and stores them for future analysis and troubleshooting. The input is various action and event data, and the output is a log file. This allows for continuous improvement and reliability of the system.

[2048] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2049] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2050] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2051] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2052] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2053] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2054] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2055] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2056] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2057] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2058] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2059] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2060] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2061] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2062] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2063] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2064] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2065] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2066] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2067] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2068] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2069] The following is further disclosed regarding the above embodiment.

[2070] (Claim 1)

[2071] a means for collecting sensor data;

[2072] A means for learning user behavior patterns using artificial intelligence based on collected sensor data;

[2073] A means for generating an action suitable for the user based on the learned behavioral pattern;

[2074] means for controlling the device to execute the generated action;

[2075] a means for notifying a caregiver upon the occurrence of a particular event;

[2076] A system that includes a means to log all actions and events.

[2077] (Claim 2)

[2078] 10. The system of claim 1, wherein the collected sensor data includes temperature data, lighting data, and door open / close status data.

[2079] (Claim 3)

[2080] The system of claim 1, which automatically turns on lights when the user wakes up in the middle of the night, maintains an appropriate temperature, and issues an alert if the user accidentally opens or closes a door, based on the learned behavioral patterns.

[2081] "Example 1"

[2082] (Claim 1)

[2083] means for collecting data from a plurality of sensors installed in the environment;

[2084] A means for learning user behavior patterns using artificial intelligence based on collected sensor data;

[2085] A means for generating an action suitable for the user based on the learned behavioral pattern;

[2086] means for controlling the device to execute the generated action;

[2087] a means of notifying a caregiver when a specific event occurs;

[2088] A system that includes a means to log all actions and events.

[2089] (Claim 2)

[2090] 10. The system of claim 1, wherein the collected sensor data includes temperature data, lighting data, and door open / close status data.

[2091] (Claim 3)

[2092] The system of claim 1, which predicts the location where the user will be at a specific time based on the learned behavioral patterns, turns on the lights at that location, maintains an appropriate room temperature, and alerts the caregiver if it detects any abnormal door operation.

[2093] "Application Example 1"

[2094] (Claim 1)

[2095] a means for collecting sensor data;

[2096] A means for learning user behavior patterns using artificial intelligence based on collected sensor data;

[2097] A means for generating an action suitable for the user based on the learned behavioral pattern;

[2098] means for controlling the device to execute the generated action;

[2099] a means of notifying an administrator when certain events occur;

[2100] a means of logging all actions and events;

[2101] A system that includes a means to automatically adjust environmental factors (temperature, lighting, etc.) within a physical store according to customer behavior patterns.

[2102] (Claim 2)

[2103] 2. The system of claim 1, wherein the collected sensor data includes temperature data, lighting data, human presence data, and door opening / closing status data.

[2104] (Claim 3)

[2105] The system of claim 1, which automatically adjusts the temperature and lighting in areas that are congested during specific times based on the learned behavioral patterns, and immediately notifies an administrator if an abnormality occurs.

[2106] "Example 2: Combining Emotion Engines"

[2107] (Claim 1)

[2108] a means for collecting sensor data;

[2109] A means for learning user behavior patterns using artificial intelligence based on collected sensor data;

[2110] A means for generating an action suitable for the user based on the learned behavioral pattern;

[2111] means for controlling the device to execute the generated action;

[2112] means for recognizing the user's emotions by analyzing the user's facial expressions, tone of voice, and biometric data;

[2113] means for generating appropriate actions based on the recognized user emotions;

[2114] a means for notifying a caregiver upon the occurrence of a particular event;

[2115] A system that includes a means to log all actions and events.

[2116] (Claim 2)

[2117] 10. The system of claim 1, wherein the collected sensor data includes temperature data, lighting data, and door open / close status data.

[2118] (Claim 3)

[2119] The system of claim 1, which, based on the learned behavioral patterns, automatically turns on lights when the user wakes up in the middle of the night, maintains an appropriate temperature, generates corresponding actions based on the recognized user emotions, and issues an alert for accidental door opening and closing.

[2120] "Application example 2 when combining emotion engines"

[2121] (Claim 1)

[2122] a means for collecting sensor data;

[2123] A means for learning user behavior patterns using artificial intelligence based on collected sensor data;

[2124] A means for generating an action suitable for the user based on the learned behavioral pattern;

[2125] means for controlling the device to execute the generated action;

[2126] a means for notifying a caregiver upon the occurrence of a particular event;

[2127] a means of logging all actions and events;

[2128] emotion recognition means for recognizing an emotion of a user;

[2129] means for adjusting actions based on the emotional state of the user;

[2130] A system including means for sending notifications to employees at a physical store regarding a user's emotional state.

[2131] (Claim 2)

[2132] 10. The system of claim 1, wherein the collected sensor data includes temperature data, lighting data, and door open / close status data.

[2133] (Claim 3)

[2134] 2. The system according to claim 1, wherein the emotion recognition means recognizes emotions by analyzing facial expression data and voice tone data of the user.

[2135] (Claim 4)

[2136] The system according to claim 1, further comprising a notification means for providing advice to employees in the physical store to take appropriate action based on the emotional state.

[2137] (Claim 5)

[2138] The system of claim 1, wherein the notification means displays a status regarding the user's emotional state in real time on the employee's smart display. [Explanation of symbols]

[2139] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting sensor data; A means for learning user behavior patterns using artificial intelligence based on collected sensor data; A means for generating an action suitable for the user based on the learned behavioral pattern; means for controlling the device to execute the generated action; a means for notifying a caregiver upon the occurrence of a particular event; A system that includes a means to log all actions and events.

2. The system of claim 1 , wherein the collected sensor data includes temperature data, lighting data, and door open / close status data.

3. The system of claim 1 , wherein the system automatically turns on lights when the user wakes up in the middle of the night, maintains an appropriate temperature, and issues an alert if the user accidentally opens or closes a door, based on the learned behavioral patterns.

Images