system

Abstract

We provide the system. [Solution] A means for receiving data related to the user's living environment, A means for analyzing the aforementioned living environment data to generate optimal automation suggestions for the user, A means for notifying the user of the generated automation proposal, A means for receiving settings changed by the user based on the aforementioned automated proposal, Means for controlling a smart home device to execute the received settings, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, there are many smart home devices, but in order to effectively use them, users need to set up and manage them by themselves. Therefore, these devices cannot be fully utilized, and often the opportunity to improve the quality of life is missed. In addition, due to the complex introduction process, many users have not been able to benefit from smart home technology. The purpose of this invention is to enable users to easily set up and manage smart home devices and facilitate automation for improving the quality of life.

Means for Solving the Problems

[0005] The smart home system according to the present invention includes the following means: means for receiving data relating to the user's living environment; means for analyzing the living environment data to generate optimal automation suggestions for the user; means for notifying the user of the generated automation suggestions; means for receiving settings changed by the user based on the notified automation suggestions; and means for controlling smart home devices to execute the received settings. By combining these means, the system provides an environment in which the user can effectively use and easily manage smart home devices. This allows the user to enjoy a smart home environment optimized for their lifestyle without requiring complex settings.

[0006] "User" refers to an individual consumer or their household that uses the system.

[0007] "Living environment data" refers to all data related to the user's life, such as camera information, daily activity records, and information about the layout of their residence.

[0008] "Receiving means" refers to a mechanism for taking in data from an external source and storing it within the system.

[0009] "Analysis methods" refer to computational and machine learning algorithms used to process received data and understand user behavior patterns and needs.

[0010] "Automation suggestions" refer to recommendations for specific actions or settings generated by analytical tools to make the user's life more efficient or comfortable.

[0011] "Notification means" refers to display devices and communication devices used to inform users of analysis results and automation suggestions.

[0012] "Means for receiving settings" refers to a mechanism for incorporating user-modified or approved settings into the system.

[0013] "Control means" refers to a mechanism that issues instructions to smart home devices based on received settings and manages the operation of those devices.

[0014] "Smart home devices" refer to various devices used in the home that can be remotely controlled or automated using communication technology.

[0015] A "machine learning algorithm" refers to a set of computational procedures that learn from past data to predict future actions or make optimal suggestions. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] System Overview

[0038] The smart home automation system of this invention improves the user's daily life by receiving and analyzing data about the user's living environment and generating optimal automation suggestions. The system is broadly composed of three entities: a server, a terminal, and a user, which work together in cooperation.

[0039] User

[0040] Users record daily activity data using smart home devices. This includes things like turning lights on / off, changing temperature settings, and recording activity with a camera, and this data is periodically sent to a server via the device.

[0041] terminal

[0042] The terminal is responsible for receiving data sent from the user and transmitting it to the server. It also has a display and interface to present automation suggestions received from the server to the user. The user can review, select, and customize the provided automation suggestions, and these selections are also transmitted to the server via the terminal.

[0043] server

[0044] The server acts as the central hub for analyzing living environment data received from users and generating automation suggestions. The received data is stored in a database and analyzed using machine learning algorithms. This allows the system to understand the user's lifestyle patterns and needs, and generate optimal automation suggestions based on this information. The generated suggestions are then notified to the user via their device. The new settings, once accepted by the user, are then returned to the server and implemented by the smart home device's control system.

[0045] Examples of automation proposals

[0046] For example, if user A wakes up at 6 AM and turns on the living room lights, the server analyzes this pattern and generates an automation suggestion such as "Automatically turn on the living room lights at 6 AM." It also suggests "Automatically turn off the living room lights" at 8 AM when user A leaves for work. This allows user A to automate simple daily tasks and live a more efficient and comfortable life.

[0047] Other features

[0048] Furthermore, the system allows for user-specific customization. For example, it supports changing settings only on specific days, and automating settings based on advanced conditions (temperature, day of the week, etc.). Users input these custom settings from their devices, and the server retrieves those settings and controls the operation of smart home devices.

[0049] Examples

[0050] As a concrete example, consider a case where user B wants to set up their air conditioner to automatically adjust its temperature according to their working hours. The system receives working hour data for each day of the week and makes a suggestion to optimize the air conditioner temperature based on that data. If user B accepts the suggestion, the server applies the setting to the smart home device, and the air conditioner temperature is automatically adjusted.

[0051] Thus, the present invention is a powerful tool for simplifying users' daily lives and improving their quality of life.

[0052] The following describes the processing flow.

[0053] Step 1:

[0054] Users record daily activity data using smart home devices. This data includes turning lights on / off, changing temperature settings, and recording activity with cameras.

[0055] Step 2:

[0056] The device receives activity data recorded by the user and periodically sends it to the server. This transmission takes place over the internet.

[0057] Step 3:

[0058] The server receives living environment data sent from the terminal. The received data is stored in a database.

[0059] Step 4:

[0060] The server analyzes the received living environment data and runs machine learning algorithms to understand the user's lifestyle patterns and needs. This analysis identifies the user's habits and specific needs.

[0061] Step 5:

[0062] The server generates automation suggestions based on the analysis results. For example, if a user has a habit of waking up at 6 AM and turning on the living room lights, the server will generate a suggestion to "automatically turn on the living room lights at 6 AM."

[0063] Step 6:

[0064] The server sends the automation suggestions it generates to the terminal.

[0065] Step 7:

[0066] The terminal displays automation suggestions received from the server to the user. The user reviews these suggestions and customizes them as needed.

[0067] Step 8:

[0068] The user accepts the automation suggestion or enters customized settings and sends them to the server via their device.

[0069] Step 9:

[0070] The server receives new settings sent by the user. These settings are stored in a database and used to control smart home devices.

[0071] Step 10:

[0072] Based on the new settings received by the server, it issues instructions to control smart home devices. For example, it can set the on / off times for lights.

[0073] Step 11:

[0074] Smart home devices operate based on instructions from a server. This automates functions such as lighting and temperature control.

[0075] Through this series of processes, the present invention makes the user's life more comfortable and simplifies the operation of smart home devices.

[0076] (Example 1)

[0077] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0078] Traditional smart home automation systems required users to manually configure individual devices, and did not offer optimal suggestions based on lifestyle patterns or needs. This resulted in increased user effort and frequency of operation, ultimately diminishing convenience.

[0079] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0080] In this invention, the server includes means for collecting data about the user's living environment, means for periodically transmitting the living environment data to the server, and means for storing and analyzing the received living environment data in a database. This makes it possible to generate optimal automation suggestions based on the user's lifestyle patterns and needs, and notify the user in a form that can be easily customized. This reduces the effort required from the user and provides a more comfortable and efficient smart home experience.

[0081] A "user" is someone who uses a smart home system, provides data about their living environment, and utilizes the automation suggestions provided by the system.

[0082] "Living environment data" refers to information collected through smart home devices, such as the user's indoor activities and device settings.

[0083] A "server" is a computer system that stores and analyzes received data, generates automation suggestions, and notifies the user.

[0084] A "database" is a place on a server where user-generated data about their living environment is organized, stored, and used for later analysis.

[0085] "Analyzing" means using data analysis algorithms based on collected living environment data to understand the user's lifestyle patterns and needs.

[0086] "Automation suggestions" are information that indicates setting changes or new operating methods to make the user's life more convenient, based on the results of an analysis of living environment data.

[0087] "Notifying" refers to the act of informing the user of the generated automation suggestions, and this is done through the notification function of a smartphone or the interface of a dedicated application.

[0088] "Customization" refers to the process by which users can modify and adjust automated suggestions they receive to suit their own needs.

[0089] "Smart home devices" refer to hardware devices such as lighting, air conditioners, and cameras in the home that automate their operation based on control instructions from a server.

[0090] A "data analysis algorithm" is a computational method used to analyze collected data, employing machine learning techniques and statistical analysis techniques.

[0091] This invention is a smart home system that collects and analyzes data related to the user's living environment and generates optimal automation suggestions based on the results. This system operates through the mutual cooperation of three entities: the user, the terminal, and the server.

[0092] User

[0093] Users are the entities that provide data about their living environment. Users use smart home devices to record daily activity data. This includes things like turning living room lights on / off, changing the air conditioner temperature, and recording activities with a camera. This data is periodically sent to the server via the device.

[0094] terminal

[0095] The terminal is responsible for receiving data sent from the user and transmitting it to the server. It also features a display or interface (e.g., a smartphone app) to present automation suggestions received from the server to the user. The user can review the provided automation suggestions and customize them as needed. This customization information is also transmitted to the server via the terminal.

[0096] server

[0097] The server has a central function of analyzing living environment data received from the user and generating automation suggestions. The server stores the received data in a database (e.g., MySQL® or PostgreSQL). Next, the stored data is analyzed using machine learning algorithms with analysis software such as Python or R. This allows the server to understand the user's living patterns and needs. Based on this, optimal automation suggestions are generated and notified to the user via their device. Once the user accepts the suggestion, the new settings are sent back to the server and reflected in the control of the smart home devices. IoT protocols such as MQTT and CoAP are used for control.

[0098] Specific example

[0099] For example, if user A wakes up at 6 AM and turns on the living room lights, the server analyzes this pattern and generates an automation suggestion to "automatically turn on the living room lights at 6 AM." It also suggests "automatically turn off the living room lights" at 8 AM when user A leaves for work. This allows user A to automate simple daily tasks, making their life more convenient.

[0100] Example of a prompt

[0101] An example of a prompt message for inputting a specific example into a generative AI model is as follows:

[0102] User B wants the air conditioner temperature to be automatically adjusted according to their working hours. We want to receive working hour data for each day of the week and then suggest an optimized temperature based on that data. Please explain in detail how this system works.

[0103] The above describes the modes for carrying out the invention. This invention is designed with the aim of making the user's life more efficient and improving their comfort.

[0104] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0105] Step 1: Data Collection

[0106] Users use smart home devices to collect daily activity data. This data includes turning lights on / off, changing air conditioner temperature settings, and recording information from cameras. For example, a user turns on the living room lights. This action changes the light status to "on," and this data is recorded on the device. The input is the user's action, and the output is the log data of that action.

[0107] Step 2: Data transmission

[0108] The terminal periodically sends collected data to the server. It uses communication methods such as Wi-Fi or Bluetooth to generate data packets and transmit them via a secure protocol (such as HTTPS). For example, the terminal sends the user's air conditioner settings data to the server. The input is the operation data stored on the terminal, and the output is the data packets sent to the server.

[0109] Step 3: Save Data

[0110] The server stores the received data in a database. The database used is a database management system such as MySQL or PostgreSQL. For example, the server stores air conditioner setting data received from a terminal in the database. The input is the received data packet, and the output is the data recorded in the database.

[0111] Step 4: Data Analysis

[0112] The server uses stored data to execute data analysis algorithms (using Python or R) and analyze the user's lifestyle patterns. Machine learning techniques are used to extract patterns and trends from the data. For example, the server analyzes the user's lighting operation data to detect a pattern where the living room lights are turned on at 6 AM every morning. The input is lifestyle environment data stored in a database, and the output is lifestyle pattern data obtained through analysis.

[0113] Step 5: Generating automation proposals

[0114] Based on the analysis results, the server generates optimal automation suggestions. For example, the server might generate a suggestion to "automatically turn on the living room lights at 6 AM." The generated suggestions are then sent to the terminal. The input is lifestyle pattern data, and the output is automation suggestions.

[0115] Step 6: User presentation of the proposal

[0116] The terminal presents the user with automation suggestions received from the server. This is done through the smartphone's notification function or a dedicated application interface. The user reviews the suggestions and customizes them as needed. For example, the terminal notifies the user of a suggestion to "turn on the living room lights at 6 AM," and the user reviews and accepts the suggestion. The input is the automation suggestion sent from the server, and the output is the user's customized data.

[0117] Step 7: Submit customization information

[0118] After the user customizes the suggestion, the device sends that information to the server. For example, if the user changes the suggestion to "Turn on the living room lights at 7 AM instead of 6 AM," that customization information will be sent. The input is the user's customization data, and the output is the configuration change data sent to the server.

[0119] Step 8: Apply the settings

[0120] The server controls smart home devices based on the received customization information. It uses IoT protocols (such as MQTT or CoAP) to apply the new settings to the devices. For example, the server automatically adjusts the air conditioner temperature based on the user's working hours. The input is the retransmitted configuration change data, and the output is the applied device settings.

[0121] Through the steps described above, optimal automation based on the user's lifestyle patterns is achieved.

[0122] (Application Example 1)

[0123] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0124] Traditional smart home automation systems primarily aimed to streamline daily life by generating optimal automation suggestions based on data about the user's living environment. However, applying this technology to physical stores has great potential to improve store operations and enhance customer satisfaction. For example, there is a need to maintain comfort within stores by automatically adjusting lighting and air conditioning, to increase purchasing intent by providing appropriate promotions based on customer behavior patterns, and to strengthen security within stores. However, such a comprehensive store management system does not yet exist, and its realization remains a challenge.

[0125] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0126] In this invention, the server includes means for receiving data relating to the user's living environment, means for analyzing the living environment data to generate optimal automation suggestions for the user, means for notifying the user of the generated automation suggestions, means for receiving settings changed by the user based on the notified automation suggestions, means for controlling an operating device for executing the received settings, means for automatically adjusting lighting and air conditioning settings based on store environment data, means for sending promotional notifications based on the customer's surroundings, and means for monitoring in-store security, detecting anomalies, and providing notifications. This makes it possible to streamline store operations, improve customer satisfaction, and provide a secure environment.

[0127] "Data related to the user's living environment" refers to all information related to the user's daily life, and specifically includes lighting usage, temperature settings, activity records, etc.

[0128] An "automation proposal" is a specific suggestion that presents the user with the most suitable automation method based on the received data.

[0129] "Means of notification" refers to communication methods or interfaces used to inform users of generated automation suggestions.

[0130] "Operating devices" refer to various devices and equipment used to execute automation settings modified by the user. Examples include smart home devices and in-store equipment.

[0131] "Store environment data" refers to information related to the conditions inside the store, including lighting brightness, foot traffic, and temperature.

[0132] "Means of sending promotional notifications based on customer location" refers to a function that sends promotional content in real time based on the customer's location information and behavioral patterns.

[0133] "Means of monitoring in-store security and detecting and notifying of anomalies" refers to a system that uses surveillance cameras and other sensors to monitor the safety of the store and issues an alert when suspicious behavior or anomalies are detected.

[0134] "Video information" refers to video stream data obtained from cameras and other video acquisition devices.

[0135] "Behavioral records" refer to information that records the movements and behavioral patterns of users and customers.

[0136] "Store layout information" refers to detailed information about the layout and arrangement of items within a store.

[0137] A "machine learning algorithm" refers to an algorithm that analyzes data, automatically learns patterns and trends, and uses that information to inform future suggestions and control measures.

[0138] This invention aims to improve the efficiency of store operations and enhance customer satisfaction by utilizing data related to the user's living environment. Specific embodiments of this invention will be described in detail below.

[0139] System Overview

[0140] The smart physical store management system of the present invention is a system composed of three components: a server, a terminal, and a user, which work together in cooperation with each other.

[0141] User roles

[0142] The user collects environmental data (temperature, lighting, foot traffic, etc.) from the store using various sensors and devices installed within the store. These devices include temperature sensors, lighting control systems, and surveillance cameras. Sensor information is periodically transmitted to a terminal.

[0143] Terminal role

[0144] The terminal receives data sent by the user and forwards it to the server. Simultaneously, it has an interface for displaying and presenting automated suggestions and notifications received from the server to the user. Through this interface, the user can review, select, and customize the suggestions.

[0145] Server Role

[0146] The server is a central device that analyzes living environment data received from users and generates optimal automation suggestions. Specifically, it performs the following processes:

[0147] (1) Analysis of received data

[0148] Received data is stored in a database and analyzed using machine learning algorithms. For example, it learns lighting usage patterns and air conditioning settings to generate optimal automation suggestions.

[0149] (2) Generation of automation proposals

[0150] Based on the analyzed data, the system generates automation suggestions. For example, it might suggest increasing the lighting and lowering the air conditioning temperature during peak hours in a store.

[0151] (3) Notification and execution

[0152] The generated automation suggestions are notified to the user via their device. If the user accepts the suggestion, the settings are returned to the server and applied to the various devices.

[0153] Other features

[0154] Automatic lighting and air conditioning management: Lighting and air conditioning settings are automatically adjusted based on foot traffic and time of day within the store.

[0155] Promotional notifications: Send promotional information and discount coupons via push notifications based on specific conditions (e.g., when the customer's device is in a specific area).

[0156] Inventory Management Assist: Analyzes inventory information in real time and suggests adding out-of-stock items.

[0157] Security monitoring: Analyzes footage from in-store cameras to automatically detect and notify of suspicious activity.

[0158] Hardware and software to be used

[0159] Hardware: Temperature sensors, lighting control systems, surveillance cameras, user interface terminals

[0160] Software: AWS® IoT Core, database system, machine learning algorithms, push notification API

[0161] Specific example

[0162] For example, at 5 PM when the store is crowded, the system automatically brightens the lights and lowers the air conditioning temperature to provide a comfortable environment. Also, if there are new promotional items, customers in specific areas will receive real-time promotional notifications such as "20% off new items!"

[0163] Example input prompts for a generative AI model

[0164] Please create a promotional notification feature for your smart physical store management system based on the following conditions:

[0165] Send push notifications to customer devices when certain conditions are met.

[0166] The system analyzes sensor data within the store and sends notifications at the optimal time.

[0167] Use Python to generate code that integrates with AWS IoT Core and its API.

[0168] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0169] Step 1:

[0170] The user receives store environment data acquired from multiple sensors (temperature sensors, lighting control systems, surveillance cameras, etc.). This includes temperature, foot traffic, lighting brightness, and video. The user sends this data to their terminal.

[0171] Inputs: Temperature, lighting, pedestrian flow, surveillance camera data

[0172] Output: Store environment data is sent to the terminal.

[0173] Step 2:

[0174] The terminal receives environmental data sent by the user and formats it for transfer to the server. The formatted data is sent to the server in real time.

[0175] Input: Store environment data submitted by the user

[0176] Output: Formatted environment data is sent to the server.

[0177] Step 3:

[0178] The server stores the received environmental data in a database and analyzes the data using machine learning algorithms. For example, it learns lighting usage patterns, air conditioning settings, and customer behavior patterns.

[0179] Input: Formatted store environment data

[0180] Output: The learning model is saved in the database as an analysis result.

[0181] Step 4:

[0182] The server generates optimal automation suggestions based on the analysis results. For example, it might suggest increasing the lighting and lowering the air conditioning temperature during busy hours.

[0183] Input: Analysis results

[0184] Output: Specific automation proposals

[0185] Step 5:

[0186] The server notifies the terminal of the generated automation proposal. The terminal receives this proposal and notifies the user through the interface.

[0187] Input: Automation proposal

[0188] Output: Suggestions notified to the terminal

[0189] Step 6:

[0190] The user reviews the automated suggestions notified via their device, changes the settings as needed, and then returns them to the server. For example, they might customize the air conditioner temperature setting from 22 degrees to 24 degrees.

[0191] Input: Automation suggestions from the terminal

[0192] Output: User-modified settings

[0193] Step 7:

[0194] The server receives the settings returned by the user and applies them to the various operating devices. Specifically, it controls smart home devices to apply the new settings.

[0195] Input: User-modified settings

[0196] Output: New settings reflected in the operating device

[0197] Step 8:

[0198] The server monitors processing results and checks security within the store, issuing alerts if it detects any unusual activity. For example, if a security camera captures suspicious behavior, it will notify the user in real time.

[0199] Input: Security data within the store

[0200] Output: Alert notification

[0201] Step 9:

[0202] The server generates promotional notifications based on the customer's location and behavioral patterns, and sends push notifications to the customer via their device.

[0203] Input: Customer location information and behavioral patterns

[0204] Output: Promotional notification sent to customer

[0205] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0206] System Overview

[0207] The smart home automation system of this invention receives and analyzes user living environment data and emotional data, and provides the user with optimal automation suggestions. This system consists of a server, a terminal, a user, and an emotional engine.

[0208] User

[0209] Users record daily activity and emotional data through smart home devices and an emotion engine. Emotional data is extracted from the user's facial expressions, voice tone, and behavior using cameras and microphones. This data is transmitted to a server via the device.

[0210] terminal

[0211] The terminal receives lifestyle and emotional data sent by the user and transmits it to the server. It also has a display and interface to present automation suggestions received from the server to the user. The user can review, select, and customize the provided automation suggestions, and these selections are also transmitted to the server via the terminal.

[0212] server

[0213] The server acts as the central hub for analyzing lifestyle and emotional data received from users and generating automation suggestions. The received data is stored in a database and analyzed using machine learning algorithms and an emotional analysis engine. This allows the system to understand the user's lifestyle patterns and emotional state, and generate optimal automation suggestions based on this information. The generated suggestions are then notified to the user via their device. The new settings, once accepted by the user, are then returned to the server and implemented by the smart home device's control system.

[0214] Emotional Engine

[0215] The emotion engine analyzes the user's facial expressions, voice tone, and behavior in real time to acquire emotional data. This allows the system to understand the user's emotional state, and this information is then sent to the server.

[0216] Examples of automation proposals

[0217] For example, if user A returns home from work and turns on the living room lights, and their facial expression is detected as tired (by the emotion engine), the server analyzes this data and generates an automated suggestion such as "set the living room lights to a warm color and play relaxation music." This suggestion automatically provides user A with a comfortable environment.

[0218] Other features

[0219] Furthermore, the system allows for customization based on the user's emotions. For example, it supports features such as setting the temperature slightly lower if the user is feeling stressed, or automation based on advanced conditions (temperature, day of the week, etc.). Users enter these customization settings from their device, and the server takes those settings and controls the operation of smart home devices.

[0220] Examples

[0221] For example, if the emotion engine detects that user B is spending more time in the living room and is experiencing stress, the server uses this data to generate a suggestion to change the lighting to a softer light and set the temperature to a comfortable level. If user B accepts the suggestion, the server applies those settings to the smart home devices and automatically adjusts the environment.

[0222] Thus, the present invention makes users' lives more comfortable and efficient, and realizes flexible automation that adapts to their emotional state.

[0223] The following describes the processing flow.

[0224] Step 1:

[0225] Users record daily activity and emotional data using smart home devices and an emotion engine. Activity data includes turning lights on / off and changing temperature settings, while emotional data includes facial expressions and voice tones captured through cameras and microphones.

[0226] Step 2:

[0227] The device receives activity and emotion data recorded by the user and periodically sends it to a server. This transmission takes place over the internet.

[0228] Step 3:

[0229] The server receives living environment data and emotional data sent from the terminal. The received data is stored in a database.

[0230] Step 4:

[0231] The server analyzes the received living environment data and emotional data. Machine learning algorithms and an emotional analysis engine are used for the analysis to identify the user's lifestyle and emotional state.

[0232] Step 5:

[0233] The server generates automation suggestions based on the analysis results. For example, it might generate a suggestion such as, "If the user is fatigued at night, change the lighting to a warm color and play relaxation music."

[0234] Step 6:

[0235] The server sends the automation suggestions it generates to the terminal.

[0236] Step 7:

[0237] The terminal displays automation suggestions received from the server to the user. The user reviews these suggestions and customizes them as needed.

[0238] Step 8:

[0239] The user accepts the automation suggestion or enters customized settings and sends them to the server via their device.

[0240] Step 9:

[0241] The server receives new settings sent by the user. These settings are stored in a database and used to control smart home devices.

[0242] Step 10:

[0243] Based on the new settings received by the server, it issues instructions to control smart home devices. For example, it can set lights to turn on / off or music playback time.

[0244] Step 11:

[0245] Smart home devices operate based on instructions from a server. This allows for automatic adjustment of lighting and temperature, providing an environment tailored to the user's emotional state.

[0246] Through this series of processes, the system of the present invention makes the user's life more comfortable and automates the operation of smart home devices in accordance with their emotional state.

[0247] (Example 2)

[0248] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0249] In recent years, home devices and systems have become smarter, and there is a growing demand for them to make users' lives more comfortable. However, many smart home systems only collect data on the user's living environment and perform simple automation, lacking the flexibility to consider the user's emotional state. As a result, it is difficult to provide a truly relaxing environment for the user. Furthermore, the ability for users to customize automation suggestions is insufficient. There is a need for smart home automation systems that can solve these problems and make users' lives more comfortable and pleasant.

[0250] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0251] In this invention, the server includes means for receiving data relating to the user's living environment and emotional state, means for analyzing the living environment and emotional data to generate optimal automation suggestions for the user, and means for notifying the user of the generated automation suggestions. This enables flexible automation suggestions that take into account the user's emotional state, and allows the user to customize and accept the suggestions. This makes it possible to provide the user with a more comfortable living environment that meets their individual needs.

[0252] "Living environment data" refers to information about the environment in which users live their daily lives, including data such as temperature, humidity, lighting conditions, and sound environment of the living space.

[0253] "Emotional data" refers to information that indicates a user's emotional state, and is extracted from data such as facial expressions, voice tone, and behavioral patterns.

[0254] "Automated suggestions" refer to specific operational instructions generated by the server as a result of analyzing the user's living environment data and emotional data, in order to optimize the user's living environment.

[0255] A "smart home device" is an internet-connected device used to control electrical appliances and equipment within a home, and includes lighting, air conditioning, music players, and other similar devices.

[0256] A "machine learning algorithm" is a mathematical method for learning patterns from large amounts of data and making predictions and classifications about the future.

[0257] An "emotion analysis engine" is software or an algorithm that analyzes collected emotional data to estimate a user's emotional state.

[0258] "Notification methods" refer to features that inform users of automation suggestions, and may include displays, voice notifications, and mobile applications.

[0259] "Customization options" refer to interfaces or functions that allow users to change or adjust suggested automation settings.

[0260] "Operation means" refers to a function that allows a server or other control device to send direct operation instructions to a smart home device.

[0261] A "database" is a system used by a server to store data on living environment and emotional data that it receives.

[0262] The smart home automation system of this invention makes and executes optimal automation suggestions based on the user's living environment and emotional state. This system mainly consists of a server, terminals, users, and an emotion analysis engine.

[0263] 1. Data Collection

[0264] Users use smart home devices (cameras, microphones, etc.) to record data about their daily activities and living environment. This includes video information, audio data, activity records, and living space information. For example, facial expressions and sounds while a user is watching TV in the living room may be collected.

[0265] 2. Data transmission

[0266] The device temporarily stores the collected data and then sends it to the server. Wireless communication technologies such as Wi-Fi and Bluetooth are used for this data transfer.

[0267] 3. Data storage

[0268] The server stores the received data in a cloud storage service (e.g., Amazon S3, Google Cloud Storage). The stored data includes timestamps and various metadata.

[0269] 4. Data Analysis

[0270] The server uses machine learning algorithms (e.g., TENSORFLOW®) and emotion analysis engines (e.g., EmotionAI) to analyze the stored data. This analysis helps understand the user's emotional state and lifestyle patterns. For example, if a user shows signs of fatigue in their facial expression or voice tone, this information is analyzed and extracted as emotion data.

[0271] 5. Automated proposal generation

[0272] The server generates optimal automation suggestions for the user based on the analyzed data. These suggestions include specific instructions, such as setting the living room lighting to a warm color and playing relaxation music.

[0273] 6. Proposal Notification and Customization

[0274] The device notifies the user of the generated automation suggestions. These notifications are delivered via the display or voice message. The user reviews the suggestions and makes changes or customizations as needed. For example, the user can change the suggested music to jazz.

[0275] 7. Implement the proposal

[0276] The server receives suggestions reviewed and customized by the user and implements them on smart home devices. Specifically, it calls the API of a smart light bulb to change the color of the light and plays specified music through a smart speaker.

[0277] Specific example

[0278] For example, when user A returns home from work and turns on the living room lights, the emotion analysis engine detects user A's tired expression. This data is sent to the server, which generates an automated suggestion to "set the living room lights to a warm color and play relaxation music." This suggestion is displayed on the device, and if user A accepts it, the server sends instructions to the smart home device, and the lighting and music are automatically adjusted.

[0279] Example of a prompt

[0280] "In a smart home system, please explain how the system adjusts the environment when the emotion engine detects that the user is tired after returning home from work."

[0281] In this way, the system of the present invention can make the user's life comfortable and efficient by considering the user's living environment and emotional state and generating and executing optimal automation proposals.

[0282] The flow of the specific process in Example 2 will be described with reference to FIG. 13.

[0283] Step 1:

[0284] The user uses smart home devices (such as cameras and microphones) to record data on daily activities and living environment. As specific operations, a smart camera records the user's expression in real time, and a smart microphone captures voice tones. The input data is video information and audio data, which are temporarily stored by the smart home device.

[0285] Step 2:

[0286] The terminal receives the temporarily stored data (video information and audio data), compresses the data, and transmits it to the server. As a specific operation, the terminal uploads the data to the cloud server using Wi-Fi. The input is the data obtained from the smart home device, and the output is the compressed data packet.

[0287] Step 3:

[0288] The server stores the received data in cloud storage (e.g., Amazon S3 or Google Cloud Storage). As a specific operation, the server assigns a timestamp to the received data and transfers the data to the specified storage bucket. The input data is the data packet transmitted from the terminal, and the output data is the data file stored in the cloud.

[0289] Step 4:

[0290] The server runs machine learning algorithms (e.g., TensorFlow) and an emotion analysis engine to analyze the stored data. Specifically, the server retrieves data from the database and inputs it into the machine learning model to analyze the user's emotional state. The analysis results include the user's emotional state, such as whether they are tired or relaxed. The input data consists of video and audio data stored in the cloud, and the output is the analyzed emotion data.

[0291] Step 5:

[0292] The server generates automated suggestions based on emotional data. Specifically, the server selects the most suitable suggestion from a template based on the analysis results and customizes it. For example, for a tired user, it might generate a suggestion to "change the lighting to a warm color and play relaxation music." The input data is the analyzed emotional data, and the output is the generated automated suggestion.

[0293] Step 6:

[0294] The terminal notifies the user of automation suggestions received from the server. Specifically, the terminal displays a pop-up notification on its screen, showing the suggestion content. The input consists of suggestion data sent from the server, and the output is a notification visible to the user.

[0295] Step 7:

[0296] The user reviews, selects, and customizes the automated suggestions they receive. Specifically, the user interacts with the terminal's interface to accept or customize suggestions. For example, they can change the relaxation music to jazz. The input data is the suggestions displayed on the terminal, and the output data is the customized suggestions.

[0297] Step 8:

[0298] The server receives customized suggestions and implements them on smart home devices. Specifically, the server calls the smart light bulb's API to change the light color and controls the smart speaker to play the specified music. The input data is the user's customized suggestions, and the output data is the settings changes made to the executed devices.

[0299] (Application Example 2)

[0300] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0301] Conventional smart home automation systems are limited to automation suggestions based on simple living environment data, making it difficult to optimize the environment while considering the emotional state of users and customers. Furthermore, in physical stores, there is a lack of means to improve the shopping experience using customer emotional data, making it challenging to improve the quality of customer service. Additionally, product recommendations and environmental adjustments are often performed manually by staff, highlighting the need for a system that can effectively assist customers.

[0302] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0303] In this invention, the server includes means for receiving data related to the user's living environment and emotional data, means for analyzing the living environment data and emotional data to generate an optimal automation proposal for the user, means for notifying the user of the generated automation proposal, means for receiving settings changed by the user based on the notified automation proposal, means for controlling a smart environment device for executing the received settings, means for detecting emotional data (expressions, voice tones, actions) of customers in a physical store, and means for analyzing the emotional data and notifying store staff of specific product proposals and environmental adjustments. Thereby, it becomes possible to optimize the living environment and shopping experience based on the emotional states of users and customers.

[0304] "Living environment data" refers to all environmental information related to the user's life and includes visual sensor information, audio sensor information, action records, floor plan information of the residence, etc.

[0305] "Emotional data" refers to information on emotional states extracted from the expressions, voice tones, actions, etc. of users and customers.

[0306] "Automation proposal" refers to an optimal environmental setting or action proposal analyzed based on living environment data and emotional data and provided to users and customers.

[0307] "Smart environment device" mainly refers to electronic devices and apparatuses for automatically controlling the environment of a home or store and includes lighting, air conditioning, music playback devices, etc.

[0308] "Emotion analysis engine" refers to an analysis system for analyzing emotional data such as the expressions, voice tones, actions of users and customers in real time to grasp the emotional state.

[0309] "Physical store" refers to a commercial facility that customers can physically visit and is a place where products are displayed and sold.

[0310] "Specific product recommendations" refer to suggesting the most suitable products and services to individual customers based on analyzed emotional data.

[0311] "Environmental adjustment" refers to automatically setting environmental elements such as lighting, music, and temperature to their optimal state based on user and customer sentiment data.

[0312] "Analysis means" refers to a combination of hardware and software for processing and analyzing received data, particularly those using machine learning algorithms or sentiment analysis engines.

[0313] "Notification means" refers to methods and devices for communicating generated automation suggestions and information to users and store staff, and includes interfaces such as smartphones and smart glasses.

[0314] System Overview

[0315] The system implementing this invention receives and analyzes user and store customer living environment data and emotional data, and generates optimal automation suggestions based on this data. The system consists of a server, terminals, users, an emotional analysis engine, and smart environment devices.

[0316] Users and customers

[0317] Users and customers collect environmental and emotional data using cameras, microphones, and other devices. This includes visual sensor information (facial expressions), voice sensor information (voice tone), and behavioral records. The collected data is transmitted to the server via the user's device.

[0318] terminal

[0319] The terminal receives user and customer data and transmits it to the server. It also has an interface for presenting automation suggestions received from the server to the user. This interface consists of devices such as smartphones and smart glasses.

[0320] server

[0321] The server is the central hub for analyzing living environment data and emotional data received from users and customers. Machine learning algorithms and an emotional analysis engine are used for the analysis. Based on the analysis results, optimal automation suggestions are generated for users and customers and notified via their devices. If a user accepts a suggestion, the settings are reflected in the smart environment device, and the environment is automatically adjusted.

[0322] Emotion analysis engine

[0323] The emotion analysis engine analyzes user and customer facial expressions, voice tone, and behavior in real time to acquire emotional data. This data is sent to a server for further analysis and used to generate automated suggestions.

[0324] Hardware and software to be used

[0325] Camera: As a visual sensor, we will use the Logitech HD Pro Webcam C920 as an example.

[0326] Microphone: As an example, we will use the Blue Yeti USB Microphone as the voice sensor.

[0327] Smartphones / smart glasses: Used as a means of notification and data collection.

[0328] Analysis software: Machine learning algorithms and sentiment analysis engines.

[0329] Specific example

[0330] For example, if a customer in a physical store is looking at a product but shows no interest (the emotion analysis engine determines this to be "lack of interest"), the server analyzes this data and notifies the staff member wearing smart glasses to "introduce the new product to customer X." This notification allows the store staff to make the most appropriate suggestions to the customer.

[0331] Example of a prompt

[0332] "emotion_data: Customer X has a dissatisfied expression, their voice tone is low, and they are not showing interest in the product."

[0333] The above describes the embodiments for carrying out this invention. This system enables optimal environmental adjustment and product recommendations based on the emotional state of users and customers.

[0334] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0335] Step 1:

[0336] Data collection from users and customers

[0337] The user collects environmental data (visual sensor information, audio sensor information, behavioral records) and emotional data using a camera and microphone. This data is transmitted to the user's device (smartphone or smart glasses).

[0338] Input: Visual sensor information, audio sensor information, and activity records collected by the camera and microphone.

[0339] Output: Living environment data and emotional data transmitted to the terminal.

[0340] Step 2:

[0341] Sending data to the server

[0342] The device sends the collected data to the server. The server receives this data and stores it in a database.

[0343] Input: Living environment data and emotional data transmitted from the device.

[0344] Output: Living environment data and emotional data stored on the server

[0345] Step 3:

[0346] Data Analysis

[0347] The server analyzes the received data using machine learning algorithms and sentiment analysis engines. This identifies the emotional state and lifestyle patterns of users and customers.

[0348] Input: Living environment data and emotional data stored on the server

[0349] Output: Analyzed emotional states and lifestyle patterns

[0350] Step 4:

[0351] Generation of automation proposals

[0352] Based on the analyzed data, the server generates optimal automation suggestions for users and customers. This includes specific product recommendations and environment adjustments.

[0353] Input: Analyzed emotional states and lifestyle patterns

[0354] Output: Generated automation proposals

[0355] Step 5:

[0356] Automation suggestion notification

[0357] The server notifies the user's device of the generated automation suggestions. The device then displays the suggestions to the user and store staff via smartphones or smart glasses.

[0358] Input: Generated automation proposals

[0359] Output: Automation suggestions displayed on the terminal

[0360] Step 6:

[0361] Received user settings change

[0362] Users and store staff change their settings based on the automated suggestions they receive. The terminal then sends these setting changes to the server.

[0363] Input: Settings changed by users and store staff

[0364] Output: Configuration changes sent to the server

[0365] Step 7:

[0366] Smart Environment Device Control

[0367] The server controls smart environment devices based on the received configuration changes. This ensures that lighting, air conditioning, music playback devices, and other devices are set to their optimal state.

[0368] Input: Configuration changes sent to the server

[0369] Output: Smart environment devices with changed settings reflected.

[0370] The above outlines the specific processing steps of this system. This makes it possible to optimize the living environment and shopping experience based on the emotional state of users and customers.

[0371] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0372] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0373] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0374] [Second Embodiment]

[0375] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0376] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0377] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0378] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0379] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0380] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0381] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0382] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0383] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0384] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0385] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0386] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0387] System Overview

[0388] The smart home automation system of this invention improves the user's daily life by receiving and analyzing data about the user's living environment and generating optimal automation suggestions. The system is broadly composed of three entities: a server, a terminal, and a user, which work together in cooperation.

[0389] User

[0390] Users record daily activity data using smart home devices. This includes things like turning lights on / off, changing temperature settings, and recording activity with a camera, and this data is periodically sent to a server via the device.

[0391] terminal

[0392] The terminal is responsible for receiving data sent from the user and transmitting it to the server. It also has a display and interface to present automation suggestions received from the server to the user. The user can review, select, and customize the provided automation suggestions, and these selections are also transmitted to the server via the terminal.

[0393] server

[0394] The server acts as the central hub for analyzing living environment data received from users and generating automation suggestions. The received data is stored in a database and analyzed using machine learning algorithms. This allows the system to understand the user's lifestyle patterns and needs, and generate optimal automation suggestions based on this information. The generated suggestions are then notified to the user via their device. The new settings, once accepted by the user, are then returned to the server and implemented by the smart home device's control system.

[0395] Examples of automation proposals

[0396] For example, if user A wakes up at 6 AM and turns on the living room lights, the server analyzes this pattern and generates an automation suggestion such as "Automatically turn on the living room lights at 6 AM." It also suggests "Automatically turn off the living room lights" at 8 AM when user A leaves for work. This allows user A to automate simple daily tasks and live a more efficient and comfortable life.

[0397] Other features

[0398] Furthermore, the system allows for user-specific customization. For example, it supports changing settings only on specific days, and automating settings based on advanced conditions (temperature, day of the week, etc.). Users input these custom settings from their devices, and the server retrieves those settings and controls the operation of smart home devices.

[0399] Examples

[0400] As a concrete example, consider a case where user B wants to set up their air conditioner to automatically adjust its temperature according to their working hours. The system receives working hour data for each day of the week and makes a suggestion to optimize the air conditioner temperature based on that data. If user B accepts the suggestion, the server applies the setting to the smart home device, and the air conditioner temperature is automatically adjusted.

[0401] Thus, the present invention is a powerful tool for simplifying users' daily lives and improving their quality of life.

[0402] The following describes the processing flow.

[0403] Step 1:

[0404] Users record daily activity data using smart home devices. This data includes turning lights on / off, changing temperature settings, and recording activity with cameras.

[0405] Step 2:

[0406] The device receives activity data recorded by the user and periodically sends it to the server. This transmission takes place over the internet.

[0407] Step 3:

[0408] The server receives living environment data sent from the terminal. The received data is stored in a database.

[0409] Step 4:

[0410] The server analyzes the received living environment data and runs machine learning algorithms to understand the user's lifestyle patterns and needs. This analysis identifies the user's habits and specific needs.

[0411] Step 5:

[0412] The server generates automation suggestions based on the analysis results. For example, if a user has a habit of waking up at 6 AM and turning on the living room lights, the server will generate a suggestion to "automatically turn on the living room lights at 6 AM."

[0413] Step 6:

[0414] The server sends the automation suggestions it generates to the terminal.

[0415] Step 7:

[0416] The terminal displays automation suggestions received from the server to the user. The user reviews these suggestions and customizes them as needed.

[0417] Step 8:

[0418] The user accepts the automation suggestion or enters customized settings and sends them to the server via their device.

[0419] Step 9:

[0420] The server receives new settings sent by the user. These settings are stored in a database and used to control smart home devices.

[0421] Step 10:

[0422] Based on the new settings received by the server, it issues instructions to control smart home devices. For example, it can set the on / off times for lights.

[0423] Step 11:

[0424] Smart home devices operate based on instructions from a server. This automates functions such as lighting and temperature control.

[0425] Through this series of processes, the present invention makes the user's life more comfortable and simplifies the operation of smart home devices.

[0426] (Example 1)

[0427] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0428] Traditional smart home automation systems required users to manually configure individual devices, and did not offer optimal suggestions based on lifestyle patterns or needs. This resulted in increased user effort and frequency of operation, ultimately diminishing convenience.

[0429] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0430] In this invention, the server includes means for collecting data about the user's living environment, means for periodically transmitting the living environment data to the server, and means for storing and analyzing the received living environment data in a database. This makes it possible to generate optimal automation suggestions based on the user's lifestyle patterns and needs, and notify the user in a form that can be easily customized. This reduces the effort required from the user and provides a more comfortable and efficient smart home experience.

[0431] A "user" is someone who uses a smart home system, provides data about their living environment, and utilizes the automation suggestions provided by the system.

[0432] "Living environment data" refers to information collected through smart home devices, such as the user's indoor activities and device settings.

[0433] A "server" is a computer system that stores and analyzes received data, generates automation suggestions, and notifies the user.

[0434] A "database" is a place on a server where user-generated data about their living environment is organized, stored, and used for later analysis.

[0435] "Analyzing" means using data analysis algorithms based on collected living environment data to understand the user's lifestyle patterns and needs.

[0436] "Automation suggestions" are information that indicates setting changes or new operating methods to make the user's life more convenient, based on the results of an analysis of living environment data.

[0437] "Notifying" refers to the act of informing the user of the generated automation suggestions, and this is done through the notification function of a smartphone or the interface of a dedicated application.

[0438] "Customization" refers to the process by which users can modify and adjust automated suggestions they receive to suit their own needs.

[0439] "Smart home devices" refer to hardware devices such as lighting, air conditioners, and cameras in the home that automate their operation based on control instructions from a server.

[0440] A "data analysis algorithm" is a computational method used to analyze collected data, employing machine learning techniques and statistical analysis techniques.

[0441] This invention is a smart home system that collects and analyzes data related to the user's living environment and generates optimal automation suggestions based on the results. This system operates through the mutual cooperation of three entities: the user, the terminal, and the server.

[0442] User

[0443] Users are the entities that provide data about their living environment. Users use smart home devices to record daily activity data. This includes things like turning living room lights on / off, changing the air conditioner temperature, and recording activities with a camera. This data is periodically sent to the server via the device.

[0444] terminal

[0445] The terminal is responsible for receiving data sent from the user and transmitting it to the server. It also features a display or interface (e.g., a smartphone app) to present automation suggestions received from the server to the user. The user can review the provided automation suggestions and customize them as needed. This customization information is also transmitted to the server via the terminal.

[0446] server

[0447] The server has a central function of analyzing living environment data received from the user and generating automation suggestions. The server stores the received data in a database (e.g., MySQL or PostgreSQL). Next, the stored data is analyzed using machine learning algorithms with analysis software such as Python or R. This allows the server to understand the user's living patterns and needs. Based on this, optimal automation suggestions are generated and notified to the user via their device. If the user accepts the suggestion, the new settings are sent back to the server and reflected in the control of the smart home devices. IoT protocols such as MQTT and CoAP are used for control.

[0448] Specific example

[0449] For example, if user A wakes up at 6 AM and turns on the living room lights, the server analyzes this pattern and generates an automation suggestion to "automatically turn on the living room lights at 6 AM." It also suggests "automatically turn off the living room lights" at 8 AM when user A leaves for work. This allows user A to automate simple daily tasks, making their life more convenient.

[0450] Example of a prompt

[0451] An example of a prompt message for inputting a specific example into a generative AI model is as follows:

[0452] User B wants the air conditioner temperature to be automatically adjusted according to their working hours. We want to receive working hour data for each day of the week and then suggest an optimized temperature based on that data. Please explain in detail how this system works.

[0453] The above describes the modes for carrying out the invention. This invention is designed with the aim of making the user's life more efficient and improving their comfort.

[0454] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0455] Step 1: Data Collection

[0456] Users use smart home devices to collect daily activity data. This data includes turning lights on / off, changing air conditioner temperature settings, and recording information from cameras. For example, a user turns on the living room lights. This action changes the light status to "on," and this data is recorded on the device. The input is the user's action, and the output is the log data of that action.

[0457] Step 2: Data transmission

[0458] The terminal periodically sends collected data to the server. It uses communication methods such as Wi-Fi or Bluetooth to generate data packets and transmit them via a secure protocol (such as HTTPS). For example, the terminal sends the user's air conditioner settings data to the server. The input is the operation data stored on the terminal, and the output is the data packets sent to the server.

[0459] Step 3: Save Data

[0460] The server stores the received data in a database. The database used is a database management system such as MySQL or PostgreSQL. For example, the server stores air conditioner setting data received from a terminal in the database. The input is the received data packet, and the output is the data recorded in the database.

[0461] Step 4: Data Analysis

[0462] The server uses stored data to execute data analysis algorithms (using Python or R) and analyze the user's lifestyle patterns. Machine learning techniques are used to extract patterns and trends from the data. For example, the server analyzes the user's lighting operation data to detect a pattern where the living room lights are turned on at 6 AM every morning. The input is lifestyle environment data stored in a database, and the output is lifestyle pattern data obtained through analysis.

[0463] Step 5: Generating automation proposals

[0464] Based on the analysis results, the server generates optimal automation suggestions. For example, the server might generate a suggestion to "automatically turn on the living room lights at 6 AM." The generated suggestions are then sent to the terminal. The input is lifestyle pattern data, and the output is automation suggestions.

[0465] Step 6: User presentation of the proposal

[0466] The terminal presents the user with automation suggestions received from the server. This is done through the smartphone's notification function or a dedicated application interface. The user reviews the suggestions and customizes them as needed. For example, the terminal notifies the user of a suggestion to "turn on the living room lights at 6 AM," and the user reviews and accepts the suggestion. The input is the automation suggestion sent from the server, and the output is the user's customized data.

[0467] Step 7: Submit customization information

[0468] After the user customizes the suggestion, the device sends that information to the server. For example, if the user changes the suggestion to "Turn on the living room lights at 7 AM instead of 6 AM," that customization information will be sent. The input is the user's customization data, and the output is the configuration change data sent to the server.

[0469] Step 8: Apply the settings

[0470] The server controls smart home devices based on the received customization information. It uses IoT protocols (such as MQTT or CoAP) to apply the new settings to the devices. For example, the server automatically adjusts the air conditioner temperature based on the user's working hours. The input is the retransmitted configuration change data, and the output is the applied device settings.

[0471] Through the steps described above, optimal automation based on the user's lifestyle patterns is achieved.

[0472] (Application Example 1)

[0473] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0474] Traditional smart home automation systems primarily aimed to streamline daily life by generating optimal automation suggestions based on data about the user's living environment. However, applying this technology to physical stores has great potential to improve store operations and enhance customer satisfaction. For example, there is a need to maintain comfort within stores by automatically adjusting lighting and air conditioning, to increase purchasing intent by providing appropriate promotions based on customer behavior patterns, and to strengthen security within stores. However, such a comprehensive store management system does not yet exist, and its realization remains a challenge.

[0475] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0476] In this invention, the server includes means for receiving data relating to the user's living environment, means for analyzing the living environment data to generate optimal automation suggestions for the user, means for notifying the user of the generated automation suggestions, means for receiving settings changed by the user based on the notified automation suggestions, means for controlling an operating device for executing the received settings, means for automatically adjusting lighting and air conditioning settings based on store environment data, means for sending promotional notifications based on the customer's surroundings, and means for monitoring in-store security, detecting anomalies, and providing notifications. This makes it possible to streamline store operations, improve customer satisfaction, and provide a secure environment.

[0477] "Data related to the user's living environment" refers to all information related to the user's daily life, and specifically includes lighting usage, temperature settings, activity records, etc.

[0478] An "automation proposal" is a specific suggestion that presents the user with the most suitable automation method based on the received data.

[0479] "Means of notification" refers to communication methods or interfaces used to inform users of generated automation suggestions.

[0480] "Operating devices" refer to various devices and equipment used to execute automation settings modified by the user. Examples include smart home devices and in-store equipment.

[0481] "Store environment data" refers to information related to the conditions inside the store, including lighting brightness, foot traffic, and temperature.

[0482] "Means of sending promotional notifications based on customer location" refers to a function that sends promotional content in real time based on the customer's location information and behavioral patterns.

[0483] "Means of monitoring in-store security and detecting and notifying of anomalies" refers to a system that uses surveillance cameras and other sensors to monitor the safety of the store and issues an alert when suspicious behavior or anomalies are detected.

[0484] "Video information" refers to video stream data obtained from cameras and other video acquisition devices.

[0485] "Behavioral records" refer to information that records the movements and behavioral patterns of users and customers.

[0486] "Store layout information" refers to detailed information about the layout and arrangement of items within a store.

[0487] A "machine learning algorithm" refers to an algorithm that analyzes data, automatically learns patterns and trends, and uses that information to inform future suggestions and control measures.

[0488] This invention aims to improve the efficiency of store operations and enhance customer satisfaction by utilizing data related to the user's living environment. Specific embodiments of this invention will be described in detail below.

[0489] System Overview

[0490] The smart physical store management system of the present invention is a system composed of three components: a server, a terminal, and a user, which work together in cooperation with each other.

[0491] User roles

[0492] The user collects environmental data (temperature, lighting, foot traffic, etc.) from the store using various sensors and devices installed within the store. These devices include temperature sensors, lighting control systems, and surveillance cameras. Sensor information is periodically transmitted to a terminal.

[0493] Terminal role

[0494] The terminal receives data sent by the user and forwards it to the server. Simultaneously, it has an interface for displaying and presenting automated suggestions and notifications received from the server to the user. Through this interface, the user can review, select, and customize the suggestions.

[0495] Server Role

[0496] The server is a central device that analyzes living environment data received from users and generates optimal automation suggestions. Specifically, it performs the following processes:

[0497] (1) Analysis of received data

[0498] Received data is stored in a database and analyzed using machine learning algorithms. For example, it learns lighting usage patterns and air conditioning settings to generate optimal automation suggestions.

[0499] (2) Generation of automation proposals

[0500] Based on the analyzed data, the system generates automation suggestions. For example, it might suggest increasing the lighting and lowering the air conditioning temperature during peak hours in a store.

[0501] (3) Notification and execution

[0502] The generated automation suggestions are notified to the user via their device. If the user accepts the suggestion, the settings are returned to the server and applied to the various devices.

[0503] Other features

[0504] Automatic lighting and air conditioning management: Lighting and air conditioning settings are automatically adjusted based on foot traffic and time of day within the store.

[0505] Promotional notifications: Send promotional information and discount coupons via push notifications based on specific conditions (e.g., when the customer's device is in a specific area).

[0506] Inventory Management Assist: Analyzes inventory information in real time and suggests adding out-of-stock items.

[0507] Security monitoring: Analyzes footage from in-store cameras to automatically detect and notify of suspicious activity.

[0508] Hardware and software to be used

[0509] Hardware: Temperature sensors, lighting control systems, surveillance cameras, user interface terminals

[0510] Software: AWS IoT Core, database system, machine learning algorithms, push notification API

[0511] Specific example

[0512] For example, at 5 PM when the store is crowded, the system automatically brightens the lights and lowers the air conditioning temperature to provide a comfortable environment. Also, if there are new promotional items, customers in specific areas will receive real-time promotional notifications such as "20% off new items!"

[0513] Example input prompts for a generative AI model

[0514] Please create a promotional notification feature for your smart physical store management system based on the following conditions:

[0515] Send push notifications to customer devices when certain conditions are met.

[0516] The system analyzes sensor data within the store and sends notifications at the optimal time.

[0517] Use Python to generate code that integrates with AWS IoT Core and its API.

[0518] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0519] Step 1:

[0520] The user receives store environment data acquired from multiple sensors (temperature sensors, lighting control systems, surveillance cameras, etc.). This includes temperature, foot traffic, lighting brightness, and video. The user sends this data to their terminal.

[0521] Inputs: Temperature, lighting, pedestrian flow, surveillance camera data

[0522] Output: Store environment data is sent to the terminal.

[0523] Step 2:

[0524] The terminal receives environmental data sent by the user and formats it for transfer to the server. The formatted data is sent to the server in real time.

[0525] Input: Store environment data submitted by the user

[0526] Output: Formatted environment data is sent to the server.

[0527] Step 3:

[0528] The server stores the received environmental data in a database and analyzes the data using machine learning algorithms. For example, it learns lighting usage patterns, air conditioning settings, and customer behavior patterns.

[0529] Input: Formatted store environment data

[0530] Output: The learning model is saved in the database as an analysis result.

[0531] Step 4:

[0532] The server generates optimal automation suggestions based on the analysis results. For example, it might suggest increasing the lighting and lowering the air conditioning temperature during busy hours.

[0533] Input: Analysis results

[0534] Output: Specific automation proposals

[0535] Step 5:

[0536] The server notifies the terminal of the generated automation proposal. The terminal receives this proposal and notifies the user through the interface.

[0537] Input: Automation proposal

[0538] Output: Suggestions notified to the terminal

[0539] Step 6:

[0540] The user reviews the automated suggestions notified via their device, changes the settings as needed, and then returns them to the server. For example, they might customize the air conditioner temperature setting from 22 degrees to 24 degrees.

[0541] Input: Automation suggestions from the terminal

[0542] Output: User-modified settings

[0543] Step 7:

[0544] The server receives the settings returned by the user and applies them to the various operating devices. Specifically, it controls smart home devices to apply the new settings.

[0545] Input: User-modified settings

[0546] Output: New settings reflected in the operating device

[0547] Step 8:

[0548] The server monitors processing results and checks security within the store, issuing alerts if it detects any unusual activity. For example, if a security camera captures suspicious behavior, it will notify the user in real time.

[0549] Input: Security data within the store

[0550] Output: Alert notification

[0551] Step 9:

[0552] The server generates promotional notifications based on the customer's location and behavioral patterns, and sends push notifications to the customer via their device.

[0553] Input: Customer location information and behavioral patterns

[0554] Output: Promotional notification sent to customer

[0555] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0556] System Overview

[0557] The smart home automation system of this invention receives and analyzes user living environment data and emotional data, and provides the user with optimal automation suggestions. This system consists of a server, a terminal, a user, and an emotional engine.

[0558] User

[0559] Users record daily activity and emotional data through smart home devices and an emotion engine. Emotional data is extracted from the user's facial expressions, voice tone, and behavior using cameras and microphones. This data is transmitted to a server via the device.

[0560] terminal

[0561] The terminal receives lifestyle and emotional data sent by the user and transmits it to the server. It also has a display and interface to present automation suggestions received from the server to the user. The user can review, select, and customize the provided automation suggestions, and these selections are also transmitted to the server via the terminal.

[0562] server

[0563] The server acts as the central hub for analyzing lifestyle and emotional data received from users and generating automation suggestions. The received data is stored in a database and analyzed using machine learning algorithms and an emotional analysis engine. This allows the system to understand the user's lifestyle patterns and emotional state, and generate optimal automation suggestions based on this information. The generated suggestions are then notified to the user via their device. The new settings, once accepted by the user, are then returned to the server and implemented by the smart home device's control system.

[0564] Emotional Engine

[0565] The emotion engine analyzes the user's facial expressions, voice tone, and behavior in real time to acquire emotional data. This allows the system to understand the user's emotional state, and this information is then sent to the server.

[0566] Examples of automation proposals

[0567] For example, if user A returns home from work and turns on the living room lights, and their facial expression is detected as tired (by the emotion engine), the server analyzes this data and generates an automated suggestion such as "set the living room lights to a warm color and play relaxation music." This suggestion automatically provides user A with a comfortable environment.

[0568] Other features

[0569] Furthermore, the system allows for customization based on the user's emotions. For example, it supports features such as setting the temperature slightly lower if the user is feeling stressed, or automation based on advanced conditions (temperature, day of the week, etc.). Users enter these customization settings from their device, and the server takes those settings and controls the operation of smart home devices.

[0570] Examples

[0571] For example, if the emotion engine detects that user B is spending more time in the living room and is experiencing stress, the server uses this data to generate a suggestion to change the lighting to a softer light and set the temperature to a comfortable level. If user B accepts the suggestion, the server applies those settings to the smart home devices and automatically adjusts the environment.

[0572] Thus, the present invention makes users' lives more comfortable and efficient, and realizes flexible automation that adapts to their emotional state.

[0573] The following describes the processing flow.

[0574] Step 1:

[0575] Users record daily activity and emotional data using smart home devices and an emotion engine. Activity data includes turning lights on / off and changing temperature settings, while emotional data includes facial expressions and voice tones captured through cameras and microphones.

[0576] Step 2:

[0577] The device receives activity and emotion data recorded by the user and periodically sends it to a server. This transmission takes place over the internet.

[0578] Step 3:

[0579] The server receives living environment data and emotional data sent from the terminal. The received data is stored in a database.

[0580] Step 4:

[0581] The server analyzes the received living environment data and emotional data. Machine learning algorithms and an emotional analysis engine are used for the analysis to identify the user's lifestyle and emotional state.

[0582] Step 5:

[0583] The server generates automation suggestions based on the analysis results. For example, it might generate a suggestion such as, "If the user is fatigued at night, change the lighting to a warm color and play relaxation music."

[0584] Step 6:

[0585] The server sends the automation suggestions it generates to the terminal.

[0586] Step 7:

[0587] The terminal displays automation suggestions received from the server to the user. The user reviews these suggestions and customizes them as needed.

[0588] Step 8:

[0589] The user accepts the automation suggestion or enters customized settings and sends them to the server via their device.

[0590] Step 9:

[0591] The server receives new settings sent by the user. These settings are stored in a database and used to control smart home devices.

[0592] Step 10:

[0593] Based on the new settings received by the server, it issues instructions to control smart home devices. For example, it can set lights to turn on / off or music playback time.

[0594] Step 11:

[0595] Smart home devices operate based on instructions from a server. This allows for automatic adjustment of lighting and temperature, providing an environment tailored to the user's emotional state.

[0596] Through this series of processes, the system of the present invention makes the user's life more comfortable and automates the operation of smart home devices in accordance with their emotional state.

[0597] (Example 2)

[0598] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0599] In recent years, home devices and systems have become smarter, and there is a growing demand for them to make users' lives more comfortable. However, many smart home systems only collect data on the user's living environment and perform simple automation, lacking the flexibility to consider the user's emotional state. As a result, it is difficult to provide a truly relaxing environment for the user. Furthermore, the ability for users to customize automation suggestions is insufficient. There is a need for smart home automation systems that can solve these problems and make users' lives more comfortable and pleasant.

[0600] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0601] In this invention, the server includes means for receiving data relating to the user's living environment and emotional state, means for analyzing the living environment and emotional data to generate optimal automation suggestions for the user, and means for notifying the user of the generated automation suggestions. This enables flexible automation suggestions that take into account the user's emotional state, and allows the user to customize and accept the suggestions. This makes it possible to provide the user with a more comfortable living environment that meets their individual needs.

[0602] "Living environment data" refers to information about the environment in which users live their daily lives, including data such as temperature, humidity, lighting conditions, and sound environment of the living space.

[0603] "Emotional data" refers to information that indicates a user's emotional state, and is extracted from data such as facial expressions, voice tone, and behavioral patterns.

[0604] "Automated suggestions" refer to specific operational instructions generated by the server as a result of analyzing the user's living environment data and emotional data, in order to optimize the user's living environment.

[0605] A "smart home device" is an internet-connected device used to control electrical appliances and equipment within a home, and includes lighting, air conditioning, music players, and other similar devices.

[0606] A "machine learning algorithm" is a mathematical method for learning patterns from large amounts of data and making predictions and classifications about the future.

[0607] An "emotion analysis engine" is software or an algorithm that analyzes collected emotional data to estimate a user's emotional state.

[0608] "Notification methods" refer to features that inform users of automation suggestions, and may include displays, voice notifications, and mobile applications.

[0609] "Customization options" refer to interfaces or functions that allow users to change or adjust suggested automation settings.

[0610] "Operation means" refers to a function that allows a server or other control device to send direct operation instructions to a smart home device.

[0611] A "database" is a system used by a server to store data on living environment and emotional data that it receives.

[0612] The smart home automation system of this invention makes and executes optimal automation suggestions based on the user's living environment and emotional state. This system mainly consists of a server, terminals, users, and an emotion analysis engine.

[0613] 1. Data Collection

[0614] Users use smart home devices (cameras, microphones, etc.) to record data about their daily activities and living environment. This includes video information, audio data, activity records, and living space information. For example, facial expressions and sounds while a user is watching TV in the living room may be collected.

[0615] 2. Data transmission

[0616] The device temporarily stores the collected data and then sends it to the server. Wireless communication technologies such as Wi-Fi and Bluetooth are used for this data transfer.

[0617] 3. Data storage

[0618] The server stores the received data in a cloud storage service (e.g., Amazon S3, Google Cloud Storage). The stored data includes timestamps and various metadata.

[0619] 4. Data Analysis

[0620] The server uses machine learning algorithms (e.g., TensorFlow) and sentiment analysis engines (e.g., EmotionAI) to analyze the stored data. This analysis helps understand the user's emotional state and lifestyle patterns. For example, if a user shows signs of fatigue in their facial expression or voice tone, this information is analyzed and extracted as sentiment data.

[0621] 5. Automated proposal generation

[0622] The server generates optimal automation suggestions for the user based on the analyzed data. These suggestions include specific instructions, such as setting the living room lighting to a warm color and playing relaxation music.

[0623] 6. Proposal Notification and Customization

[0624] The device notifies the user of the generated automation suggestions. These notifications are delivered via the display or voice message. The user reviews the suggestions and makes changes or customizations as needed. For example, the user can change the suggested music to jazz.

[0625] 7. Implement the proposal

[0626] The server receives suggestions reviewed and customized by the user and implements them on smart home devices. Specifically, it calls the API of a smart light bulb to change the color of the light and plays specified music through a smart speaker.

[0627] Specific example

[0628] For example, when user A returns home from work and turns on the living room lights, the emotion analysis engine detects user A's tired expression. This data is sent to the server, which generates an automated suggestion to "set the living room lights to a warm color and play relaxation music." This suggestion is displayed on the device, and if user A accepts it, the server sends instructions to the smart home device, and the lighting and music are automatically adjusted.

[0629] Example of a prompt

[0630] "In a smart home system, please explain how the system adjusts the environment when the emotion engine detects that the user is tired after returning home from work."

[0631] Thus, the system of the present invention can make the user's life more comfortable and efficient by considering the user's living environment and emotional state, and by generating and executing optimal automation suggestions.

[0632] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0633] Step 1:

[0634] Users record data about their daily activities and living environment using smart home devices (cameras, microphones, etc.). Specifically, a smart camera records the user's facial expressions in real time, and a smart microphone captures their voice tone. The input data consists of video and audio information, which is temporarily stored by the smart home devices.

[0635] Step 2:

[0636] The device receives temporarily stored data (video and audio data), compresses the data, and sends it to the server. Specifically, the device uploads the data to the cloud server using Wi-Fi. The input is data acquired from the smart home device, and the output is compressed data packets.

[0637] Step 3:

[0638] The server saves the received data to cloud storage (e.g., Amazon S3 or Google Cloud Storage). Specifically, the server adds a timestamp to the received data and transfers it to the specified storage bucket. The input data is the data packets sent from the terminal, and the output data is the data file stored in the cloud.

[0639] Step 4:

[0640] The server runs machine learning algorithms (e.g., TensorFlow) and an emotion analysis engine to analyze the stored data. Specifically, the server retrieves data from the database and inputs it into the machine learning model to analyze the user's emotional state. The analysis results include the user's emotional state, such as whether they are tired or relaxed. The input data consists of video and audio data stored in the cloud, and the output is the analyzed emotion data.

[0641] Step 5:

[0642] The server generates automated suggestions based on emotional data. Specifically, the server selects the most suitable suggestion from a template based on the analysis results and customizes it. For example, for a tired user, it might generate a suggestion to "change the lighting to a warm color and play relaxation music." The input data is the analyzed emotional data, and the output is the generated automated suggestion.

[0643] Step 6:

[0644] The terminal notifies the user of automation suggestions received from the server. Specifically, the terminal displays a pop-up notification on its screen, showing the suggestion content. The input consists of suggestion data sent from the server, and the output is a notification visible to the user.

[0645] Step 7:

[0646] The user reviews, selects, and customizes the automated suggestions they receive. Specifically, the user interacts with the terminal's interface to accept or customize suggestions. For example, they can change the relaxation music to jazz. The input data is the suggestions displayed on the terminal, and the output data is the customized suggestions.

[0647] Step 8:

[0648] The server receives customized suggestions and implements them on smart home devices. Specifically, the server calls the smart light bulb's API to change the light color and controls the smart speaker to play the specified music. The input data is the user's customized suggestions, and the output data is the settings changes made to the executed devices.

[0649] (Application Example 2)

[0650] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0651] Conventional smart home automation systems are limited to automation suggestions based on simple living environment data, making it difficult to optimize the environment while considering the emotional state of users and customers. Furthermore, in physical stores, there is a lack of means to improve the shopping experience using customer emotional data, making it challenging to improve the quality of customer service. Additionally, product recommendations and environmental adjustments are often performed manually by staff, highlighting the need for a system that can effectively assist customers.

[0652] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0653] In this invention, the server includes means for receiving data on the user's living environment and emotional data; means for analyzing the living environment data and emotional data to generate optimal automation suggestions for the user; means for notifying the user of the generated automation suggestions; means for receiving settings changed by the user based on the notified automation suggestions; means for controlling smart environment devices to execute the received settings; means for detecting customer emotional data (facial expressions, voice tone, behavior) within a physical store; and means for analyzing the emotional data and notifying store staff of specific product suggestions or environmental adjustments. This makes it possible to optimize the living environment and shopping experience based on the emotional state of the user and customers.

[0654] "Living environment data" refers to all environmental information related to the user's life, including visual sensor information, audio sensor information, activity records, and information on the layout of the residence.

[0655] "Emotional data" refers to information about the emotional state of users and customers, extracted from their facial expressions, voice tone, behavior, and other factors.

[0656] "Automated suggestions" refer to suggestions for optimal environmental settings and actions provided to users and customers, based on analysis of living environment data and emotional data.

[0657] "Smart environment devices" refer primarily to electronic devices and equipment used to automatically control the environment of homes and stores, and include lighting, air conditioning, and music playback devices.

[0658] An "emotion analysis engine" refers to an analysis system that analyzes emotional data such as facial expressions, voice tone, and behavior of users and customers in real time to understand their emotional state.

[0659] A "physical store" refers to a commercial facility that customers can physically visit, where goods are displayed and sold.

[0660] "Specific product recommendations" refer to suggesting the most suitable products and services to individual customers based on analyzed emotional data.

[0661] "Environmental adjustment" refers to automatically setting environmental elements such as lighting, music, and temperature to their optimal state based on user and customer sentiment data.

[0662] "Analysis means" refers to a combination of hardware and software for processing and analyzing received data, particularly those using machine learning algorithms or sentiment analysis engines.

[0663] "Notification means" refers to methods and devices for communicating generated automation suggestions and information to users and store staff, and includes interfaces such as smartphones and smart glasses.

[0664] System Overview

[0665] The system implementing this invention receives and analyzes user and store customer living environment data and emotional data, and generates optimal automation suggestions based on this data. The system consists of a server, terminals, users, an emotional analysis engine, and smart environment devices.

[0666] Users and customers

[0667] Users and customers collect environmental and emotional data using cameras, microphones, and other devices. This includes visual sensor information (facial expressions), voice sensor information (voice tone), and behavioral records. The collected data is transmitted to the server via the user's device.

[0668] terminal

[0669] The terminal receives user and customer data and transmits it to the server. It also has an interface for presenting automation suggestions received from the server to the user. This interface consists of devices such as smartphones and smart glasses.

[0670] server

[0671] The server is the central hub for analyzing living environment data and emotional data received from users and customers. Machine learning algorithms and an emotional analysis engine are used for the analysis. Based on the analysis results, optimal automation suggestions are generated for users and customers and notified via their devices. If a user accepts a suggestion, the settings are reflected in the smart environment device, and the environment is automatically adjusted.

[0672] Emotion analysis engine

[0673] The emotion analysis engine analyzes user and customer facial expressions, voice tone, and behavior in real time to acquire emotional data. This data is sent to a server for further analysis and used to generate automated suggestions.

[0674] Hardware and software to be used

[0675] Camera: As a visual sensor, we will use the Logitech HD Pro Webcam C920 as an example.

[0676] Microphone: As an example, we will use the Blue Yeti USB Microphone as the voice sensor.

[0677] Smartphones / smart glasses: Used as a means of notification and data collection.

[0678] Analysis software: Machine learning algorithms and sentiment analysis engines.

[0679] Specific example

[0680] For example, if a customer in a physical store is looking at a product but shows no interest (the emotion analysis engine determines this to be "lack of interest"), the server analyzes this data and notifies the staff member wearing smart glasses to "introduce the new product to customer X." This notification allows the store staff to make the most appropriate suggestions to the customer.

[0681] Example of a prompt

[0682] "emotion_data: Customer X has a dissatisfied expression, their voice tone is low, and they are not showing interest in the product."

[0683] The above describes the embodiments for carrying out this invention. This system enables optimal environmental adjustment and product recommendations based on the emotional state of users and customers.

[0684] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0685] Step 1:

[0686] Data collection from users and customers

[0687] The user collects environmental data (visual sensor information, audio sensor information, behavioral records) and emotional data using a camera and microphone. This data is transmitted to the user's device (smartphone or smart glasses).

[0688] Input: Visual sensor information, audio sensor information, and activity records collected by the camera and microphone.

[0689] Output: Living environment data and emotional data transmitted to the terminal.

[0690] Step 2:

[0691] Sending data to the server

[0692] The device sends the collected data to the server. The server receives this data and stores it in a database.

[0693] Input: Living environment data and emotional data transmitted from the device.

[0694] Output: Living environment data and emotional data stored on the server

[0695] Step 3:

[0696] Data Analysis

[0697] The server analyzes the received data using machine learning algorithms and sentiment analysis engines. This identifies the emotional state and lifestyle patterns of users and customers.

[0698] Input: Living environment data and emotional data stored on the server

[0699] Output: Analyzed emotional states and lifestyle patterns

[0700] Step 4:

[0701] Generation of automation proposals

[0702] Based on the analyzed data, the server generates optimal automation suggestions for users and customers. This includes specific product recommendations and environment adjustments.

[0703] Input: Analyzed emotional states and lifestyle patterns

[0704] Output: Generated automation proposals

[0705] Step 5:

[0706] Automation suggestion notification

[0707] The server notifies the user's device of the generated automation suggestions. The device then displays the suggestions to the user and store staff via smartphones or smart glasses.

[0708] Input: Generated automation proposals

[0709] Output: Automation suggestions displayed on the terminal

[0710] Step 6:

[0711] Received user settings change

[0712] Users and store staff change their settings based on the automated suggestions they receive. The terminal then sends these setting changes to the server.

[0713] Input: Settings changed by users and store staff

[0714] Output: Configuration changes sent to the server

[0715] Step 7:

[0716] Smart Environment Device Control

[0717] The server controls smart environment devices based on the received configuration changes. This ensures that lighting, air conditioning, music playback devices, and other devices are set to their optimal state.

[0718] Input: Configuration changes sent to the server

[0719] Output: Smart environment devices with changed settings reflected.

[0720] The above outlines the specific processing steps of this system. This makes it possible to optimize the living environment and shopping experience based on the emotional state of users and customers.

[0721] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0722] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0723] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0724] [Third Embodiment]

[0725] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0726] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0727] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0728] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0729] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0730] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0731] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0732] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0733] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0734] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0735] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0736] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0737] System Overview

[0738] The smart home automation system of this invention improves the user's daily life by receiving and analyzing data about the user's living environment and generating optimal automation suggestions. The system is broadly composed of three entities: a server, a terminal, and a user, which work together in cooperation.

[0739] User

[0740] Users record daily activity data using smart home devices. This includes things like turning lights on / off, changing temperature settings, and recording activity with a camera, and this data is periodically sent to a server via the device.

[0741] terminal

[0742] The terminal is responsible for receiving data sent from the user and transmitting it to the server. It also has a display and interface to present automation suggestions received from the server to the user. The user can review, select, and customize the provided automation suggestions, and these selections are also transmitted to the server via the terminal.

[0743] server

[0744] The server acts as the central hub for analyzing living environment data received from users and generating automation suggestions. The received data is stored in a database and analyzed using machine learning algorithms. This allows the system to understand the user's lifestyle patterns and needs, and generate optimal automation suggestions based on this information. The generated suggestions are then notified to the user via their device. The new settings, once accepted by the user, are then returned to the server and implemented by the smart home device's control system.

[0745] Examples of automation proposals

[0746] For example, if user A wakes up at 6 AM and turns on the living room lights, the server analyzes this pattern and generates an automation suggestion such as "Automatically turn on the living room lights at 6 AM." It also suggests "Automatically turn off the living room lights" at 8 AM when user A leaves for work. This allows user A to automate simple daily tasks and live a more efficient and comfortable life.

[0747] Other features

[0748] Furthermore, the system allows for user-specific customization. For example, it supports changing settings only on specific days, and automating settings based on advanced conditions (temperature, day of the week, etc.). Users input these custom settings from their devices, and the server retrieves those settings and controls the operation of smart home devices.

[0749] Examples

[0750] As a concrete example, consider a case where user B wants to set up their air conditioner to automatically adjust its temperature according to their working hours. The system receives working hour data for each day of the week and makes a suggestion to optimize the air conditioner temperature based on that data. If user B accepts the suggestion, the server applies the setting to the smart home device, and the air conditioner temperature is automatically adjusted.

[0751] Thus, the present invention is a powerful tool for simplifying users' daily lives and improving their quality of life.

[0752] The following describes the processing flow.

[0753] Step 1:

[0754] Users record daily activity data using smart home devices. This data includes turning lights on / off, changing temperature settings, and recording activity with cameras.

[0755] Step 2:

[0756] The device receives activity data recorded by the user and periodically sends it to the server. This transmission takes place over the internet.

[0757] Step 3:

[0758] The server receives living environment data sent from the terminal. The received data is stored in a database.

[0759] Step 4:

[0760] The server analyzes the received living environment data and runs machine learning algorithms to understand the user's lifestyle patterns and needs. This analysis identifies the user's habits and specific needs.

[0761] Step 5:

[0762] The server generates automation suggestions based on the analysis results. For example, if a user has a habit of waking up at 6 AM and turning on the living room lights, the server will generate a suggestion to "automatically turn on the living room lights at 6 AM."

[0763] Step 6:

[0764] The server sends the automation suggestions it generates to the terminal.

[0765] Step 7:

[0766] The terminal displays automation suggestions received from the server to the user. The user reviews these suggestions and customizes them as needed.

[0767] Step 8:

[0768] The user accepts the automation suggestion or enters customized settings and sends them to the server via their device.

[0769] Step 9:

[0770] The server receives new settings sent by the user. These settings are stored in a database and used to control smart home devices.

[0771] Step 10:

[0772] Based on the new settings received by the server, it issues instructions to control smart home devices. For example, it can set the on / off times for lights.

[0773] Step 11:

[0774] Smart home devices operate based on instructions from a server. This automates functions such as lighting and temperature control.

[0775] Through this series of processes, the present invention makes the user's life more comfortable and simplifies the operation of smart home devices.

[0776] (Example 1)

[0777] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0778] Traditional smart home automation systems required users to manually configure individual devices, and did not offer optimal suggestions based on lifestyle patterns or needs. This resulted in increased user effort and frequency of operation, ultimately diminishing convenience.

[0779] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0780] In this invention, the server includes means for collecting data about the user's living environment, means for periodically transmitting the living environment data to the server, and means for storing and analyzing the received living environment data in a database. This makes it possible to generate optimal automation suggestions based on the user's lifestyle patterns and needs, and notify the user in a form that can be easily customized. This reduces the effort required from the user and provides a more comfortable and efficient smart home experience.

[0781] A "user" is someone who uses a smart home system, provides data about their living environment, and utilizes the automation suggestions provided by the system.

[0782] "Living environment data" refers to information collected through smart home devices, such as the user's indoor activities and device settings.

[0783] A "server" is a computer system that stores and analyzes received data, generates automation suggestions, and notifies the user.

[0784] A "database" is a place on a server where user-generated data about their living environment is organized, stored, and used for later analysis.

[0785] "Analyzing" means using data analysis algorithms based on collected living environment data to understand the user's lifestyle patterns and needs.

[0786] "Automation suggestions" are information that indicates setting changes or new operating methods to make the user's life more convenient, based on the results of an analysis of living environment data.

[0787] "Notifying" refers to the act of informing the user of the generated automation suggestions, and this is done through the notification function of a smartphone or the interface of a dedicated application.

[0788] "Customization" refers to the process by which users can modify and adjust automated suggestions they receive to suit their own needs.

[0789] "Smart home devices" refer to hardware devices such as lighting, air conditioners, and cameras in the home that automate their operation based on control instructions from a server.

[0790] A "data analysis algorithm" is a computational method used to analyze collected data, employing machine learning techniques and statistical analysis techniques.

[0791] This invention is a smart home system that collects and analyzes data related to the user's living environment and generates optimal automation suggestions based on the results. This system operates through the mutual cooperation of three entities: the user, the terminal, and the server.

[0792] User

[0793] Users are the entities that provide data about their living environment. Users use smart home devices to record daily activity data. This includes things like turning living room lights on / off, changing the air conditioner temperature, and recording activities with a camera. This data is periodically sent to the server via the device.

[0794] terminal

[0795] The terminal is responsible for receiving data sent from the user and transmitting it to the server. It also features a display or interface (e.g., a smartphone app) to present automation suggestions received from the server to the user. The user can review the provided automation suggestions and customize them as needed. This customization information is also transmitted to the server via the terminal.

[0796] server

[0797] The server has a central function of analyzing living environment data received from the user and generating automation suggestions. The server stores the received data in a database (e.g., MySQL or PostgreSQL). Next, the stored data is analyzed using machine learning algorithms with analysis software such as Python or R. This allows the server to understand the user's living patterns and needs. Based on this, optimal automation suggestions are generated and notified to the user via their device. If the user accepts the suggestion, the new settings are sent back to the server and reflected in the control of the smart home devices. IoT protocols such as MQTT and CoAP are used for control.

[0798] Specific example

[0799] For example, if user A wakes up at 6 AM and turns on the living room lights, the server analyzes this pattern and generates an automation suggestion to "automatically turn on the living room lights at 6 AM." It also suggests "automatically turn off the living room lights" at 8 AM when user A leaves for work. This allows user A to automate simple daily tasks, making their life more convenient.

[0800] Example of a prompt

[0801] An example of a prompt message for inputting a specific example into a generative AI model is as follows:

[0802] User B wants the air conditioner temperature to be automatically adjusted according to their working hours. We want to receive working hour data for each day of the week and then suggest an optimized temperature based on that data. Please explain in detail how this system works.

[0803] The above describes the modes for carrying out the invention. This invention is designed with the aim of making the user's life more efficient and improving their comfort.

[0804] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0805] Step 1: Data Collection

[0806] Users use smart home devices to collect daily activity data. This data includes turning lights on / off, changing air conditioner temperature settings, and recording information from cameras. For example, a user turns on the living room lights. This action changes the light status to "on," and this data is recorded on the device. The input is the user's action, and the output is the log data of that action.

[0807] Step 2: Data transmission

[0808] The terminal periodically sends collected data to the server. It uses communication methods such as Wi-Fi or Bluetooth to generate data packets and transmit them via a secure protocol (such as HTTPS). For example, the terminal sends the user's air conditioner settings data to the server. The input is the operation data stored on the terminal, and the output is the data packets sent to the server.

[0809] Step 3: Save Data

[0810] The server stores the received data in a database. The database used is a database management system such as MySQL or PostgreSQL. For example, the server stores air conditioner setting data received from a terminal in the database. The input is the received data packet, and the output is the data recorded in the database.

[0811] Step 4: Data Analysis

[0812] The server uses stored data to execute data analysis algorithms (using Python or R) and analyze the user's lifestyle patterns. Machine learning techniques are used to extract patterns and trends from the data. For example, the server analyzes the user's lighting operation data to detect a pattern where the living room lights are turned on at 6 AM every morning. The input is lifestyle environment data stored in a database, and the output is lifestyle pattern data obtained through analysis.

[0813] Step 5: Generating automation proposals

[0814] Based on the analysis results, the server generates optimal automation suggestions. For example, the server might generate a suggestion to "automatically turn on the living room lights at 6 AM." The generated suggestions are then sent to the terminal. The input is lifestyle pattern data, and the output is automation suggestions.

[0815] Step 6: User presentation of the proposal

[0816] The terminal presents the user with automation suggestions received from the server. This is done through the smartphone's notification function or a dedicated application interface. The user reviews the suggestions and customizes them as needed. For example, the terminal notifies the user of a suggestion to "turn on the living room lights at 6 AM," and the user reviews and accepts the suggestion. The input is the automation suggestion sent from the server, and the output is the user's customized data.

[0817] Step 7: Submit customization information

[0818] After the user customizes the suggestion, the device sends that information to the server. For example, if the user changes the suggestion to "Turn on the living room lights at 7 AM instead of 6 AM," that customization information will be sent. The input is the user's customization data, and the output is the configuration change data sent to the server.

[0819] Step 8: Apply the settings

[0820] The server controls smart home devices based on the received customization information. It uses IoT protocols (such as MQTT or CoAP) to apply the new settings to the devices. For example, the server automatically adjusts the air conditioner temperature based on the user's working hours. The input is the retransmitted configuration change data, and the output is the applied device settings.

[0821] Through the steps described above, optimal automation based on the user's lifestyle patterns is achieved.

[0822] (Application Example 1)

[0823] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0824] Traditional smart home automation systems primarily aimed to streamline daily life by generating optimal automation suggestions based on data about the user's living environment. However, applying this technology to physical stores has great potential to improve store operations and enhance customer satisfaction. For example, there is a need to maintain comfort within stores by automatically adjusting lighting and air conditioning, to increase purchasing intent by providing appropriate promotions based on customer behavior patterns, and to strengthen security within stores. However, such a comprehensive store management system does not yet exist, and its realization remains a challenge.

[0825] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0826] In this invention, the server includes means for receiving data relating to the user's living environment, means for analyzing the living environment data to generate optimal automation suggestions for the user, means for notifying the user of the generated automation suggestions, means for receiving settings changed by the user based on the notified automation suggestions, means for controlling an operating device for executing the received settings, means for automatically adjusting lighting and air conditioning settings based on store environment data, means for sending promotional notifications based on the customer's surroundings, and means for monitoring in-store security, detecting anomalies, and providing notifications. This makes it possible to streamline store operations, improve customer satisfaction, and provide a secure environment.

[0827] "Data related to the user's living environment" refers to all information related to the user's daily life, and specifically includes lighting usage, temperature settings, activity records, etc.

[0828] An "automation proposal" is a specific suggestion that presents the user with the most suitable automation method based on the received data.

[0829] "Means of notification" refers to communication methods or interfaces used to inform users of generated automation suggestions.

[0830] "Operating devices" refer to various devices and equipment used to execute automation settings modified by the user. Examples include smart home devices and in-store equipment.

[0831] "Store environment data" refers to information related to the conditions inside the store, including lighting brightness, foot traffic, and temperature.

[0832] "Means of sending promotional notifications based on customer location" refers to a function that sends promotional content in real time based on the customer's location information and behavioral patterns.

[0833] "Means of monitoring in-store security and detecting and notifying of anomalies" refers to a system that uses surveillance cameras and other sensors to monitor the safety of the store and issues an alert when suspicious behavior or anomalies are detected.

[0834] "Video information" refers to video stream data obtained from cameras and other video acquisition devices.

[0835] "Behavioral records" refer to information that records the movements and behavioral patterns of users and customers.

[0836] "Store layout information" refers to detailed information about the layout and arrangement of items within a store.

[0837] A "machine learning algorithm" refers to an algorithm that analyzes data, automatically learns patterns and trends, and uses that information to inform future suggestions and control measures.

[0838] This invention aims to improve the efficiency of store operations and enhance customer satisfaction by utilizing data related to the user's living environment. Specific embodiments of this invention will be described in detail below.

[0839] System Overview

[0840] The smart physical store management system of the present invention is a system composed of three components: a server, a terminal, and a user, which work together in cooperation with each other.

[0841] User roles

[0842] The user collects environmental data (temperature, lighting, foot traffic, etc.) from the store using various sensors and devices installed within the store. These devices include temperature sensors, lighting control systems, and surveillance cameras. Sensor information is periodically transmitted to a terminal.

[0843] Terminal role

[0844] The terminal receives data sent by the user and forwards it to the server. Simultaneously, it has an interface for displaying and presenting automated suggestions and notifications received from the server to the user. Through this interface, the user can review, select, and customize the suggestions.

[0845] Server Role

[0846] The server is a central device that analyzes living environment data received from users and generates optimal automation suggestions. Specifically, it performs the following processes:

[0847] (1) Analysis of received data

[0848] Received data is stored in a database and analyzed using machine learning algorithms. For example, it learns lighting usage patterns and air conditioning settings to generate optimal automation suggestions.

[0849] (2) Generation of automation proposals

[0850] Based on the analyzed data, the system generates automation suggestions. For example, it might suggest increasing the lighting and lowering the air conditioning temperature during peak hours in a store.

[0851] (3) Notification and execution

[0852] The generated automation suggestions are notified to the user via their device. If the user accepts the suggestion, the settings are returned to the server and applied to the various devices.

[0853] Other features

[0854] Automatic lighting and air conditioning management: Lighting and air conditioning settings are automatically adjusted based on foot traffic and time of day within the store.

[0855] Promotional notifications: Send promotional information and discount coupons via push notifications based on specific conditions (e.g., when the customer's device is in a specific area).

[0856] Inventory Management Assist: Analyzes inventory information in real time and suggests adding out-of-stock items.

[0857] Security monitoring: Analyzes footage from in-store cameras to automatically detect and notify of suspicious activity.

[0858] Hardware and software to be used

[0859] Hardware: Temperature sensors, lighting control systems, surveillance cameras, user interface terminals

[0860] Software: AWS IoT Core, database system, machine learning algorithms, push notification API

[0861] Specific example

[0862] For example, at 5 PM when the store is crowded, the system automatically brightens the lights and lowers the air conditioning temperature to provide a comfortable environment. Also, if there are new promotional items, customers in specific areas will receive real-time promotional notifications such as "20% off new items!"

[0863] Example input prompts for a generative AI model

[0864] Please create a promotional notification feature for your smart physical store management system based on the following conditions:

[0865] Send push notifications to customer devices when certain conditions are met.

[0866] The system analyzes sensor data within the store and sends notifications at the optimal time.

[0867] Use Python to generate code that integrates with AWS IoT Core and its API.

[0868] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0869] Step 1:

[0870] The user receives store environment data acquired from multiple sensors (temperature sensors, lighting control systems, surveillance cameras, etc.). This includes temperature, foot traffic, lighting brightness, and video. The user sends this data to their terminal.

[0871] Inputs: Temperature, lighting, pedestrian flow, surveillance camera data

[0872] Output: Store environment data is sent to the terminal.

[0873] Step 2:

[0874] The terminal receives environmental data sent by the user and formats it for transfer to the server. The formatted data is sent to the server in real time.

[0875] Input: Store environment data submitted by the user

[0876] Output: Formatted environment data is sent to the server.

[0877] Step 3:

[0878] The server stores the received environmental data in a database and analyzes the data using machine learning algorithms. For example, it learns lighting usage patterns, air conditioning settings, and customer behavior patterns.

[0879] Input: Formatted store environment data

[0880] Output: The learning model is saved in the database as an analysis result.

[0881] Step 4:

[0882] The server generates optimal automation suggestions based on the analysis results. For example, it might suggest increasing the lighting and lowering the air conditioning temperature during busy hours.

[0883] Input: Analysis results

[0884] Output: Specific automation proposals

[0885] Step 5:

[0886] The server notifies the terminal of the generated automation proposal. The terminal receives this proposal and notifies the user through the interface.

[0887] Input: Automation proposal

[0888] Output: Suggestions notified to the terminal

[0889] Step 6:

[0890] The user reviews the automated suggestions notified via their device, changes the settings as needed, and then returns them to the server. For example, they might customize the air conditioner temperature setting from 22 degrees to 24 degrees.

[0891] Input: Automation suggestions from the terminal

[0892] Output: User-modified settings

[0893] Step 7:

[0894] The server receives the settings returned by the user and applies them to the various operating devices. Specifically, it controls smart home devices to apply the new settings.

[0895] Input: User-modified settings

[0896] Output: New settings reflected in the operating device

[0897] Step 8:

[0898] The server monitors processing results and checks security within the store, issuing alerts if it detects any unusual activity. For example, if a security camera captures suspicious behavior, it will notify the user in real time.

[0899] Input: Security data within the store

[0900] Output: Alert notification

[0901] Step 9:

[0902] The server generates promotional notifications based on the customer's location and behavioral patterns, and sends push notifications to the customer via their device.

[0903] Input: Customer location information and behavioral patterns

[0904] Output: Promotional notification sent to customer

[0905] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0906] System Overview

[0907] The smart home automation system of this invention receives and analyzes user living environment data and emotional data, and provides the user with optimal automation suggestions. This system consists of a server, a terminal, a user, and an emotional engine.

[0908] User

[0909] Users record daily activity and emotional data through smart home devices and an emotion engine. Emotional data is extracted from the user's facial expressions, voice tone, and behavior using cameras and microphones. This data is transmitted to a server via the device.

[0910] terminal

[0911] The terminal receives lifestyle and emotional data sent by the user and transmits it to the server. It also has a display and interface to present automation suggestions received from the server to the user. The user can review, select, and customize the provided automation suggestions, and these selections are also transmitted to the server via the terminal.

[0912] server

[0913] The server acts as the central hub for analyzing lifestyle and emotional data received from users and generating automation suggestions. The received data is stored in a database and analyzed using machine learning algorithms and an emotional analysis engine. This allows the system to understand the user's lifestyle patterns and emotional state, and generate optimal automation suggestions based on this information. The generated suggestions are then notified to the user via their device. The new settings, once accepted by the user, are then returned to the server and implemented by the smart home device's control system.

[0914] Emotional Engine

[0915] The emotion engine analyzes the user's facial expressions, voice tone, and behavior in real time to acquire emotional data. This allows the system to understand the user's emotional state, and this information is then sent to the server.

[0916] Examples of automation proposals

[0917] For example, if user A returns home from work and turns on the living room lights, and their facial expression is detected as tired (by the emotion engine), the server analyzes this data and generates an automated suggestion such as "set the living room lights to a warm color and play relaxation music." This suggestion automatically provides user A with a comfortable environment.

[0918] Other features

[0919] Furthermore, the system allows for customization based on the user's emotions. For example, it supports features such as setting the temperature slightly lower if the user is feeling stressed, or automation based on advanced conditions (temperature, day of the week, etc.). Users enter these customization settings from their device, and the server takes those settings and controls the operation of smart home devices.

[0920] Examples

[0921] For example, if the emotion engine detects that user B is spending more time in the living room and is experiencing stress, the server uses this data to generate a suggestion to change the lighting to a softer light and set the temperature to a comfortable level. If user B accepts the suggestion, the server applies those settings to the smart home devices and automatically adjusts the environment.

[0922] Thus, the present invention makes users' lives more comfortable and efficient, and realizes flexible automation that adapts to their emotional state.

[0923] The following describes the processing flow.

[0924] Step 1:

[0925] Users record daily activity and emotional data using smart home devices and an emotion engine. Activity data includes turning lights on / off and changing temperature settings, while emotional data includes facial expressions and voice tones captured through cameras and microphones.

[0926] Step 2:

[0927] The device receives activity and emotion data recorded by the user and periodically sends it to a server. This transmission takes place over the internet.

[0928] Step 3:

[0929] The server receives living environment data and emotional data sent from the terminal. The received data is stored in a database.

[0930] Step 4:

[0931] The server analyzes the received living environment data and emotional data. Machine learning algorithms and an emotional analysis engine are used for the analysis to identify the user's lifestyle and emotional state.

[0932] Step 5:

[0933] The server generates automation suggestions based on the analysis results. For example, it might generate a suggestion such as, "If the user is fatigued at night, change the lighting to a warm color and play relaxation music."

[0934] Step 6:

[0935] The server sends the automation suggestions it generates to the terminal.

[0936] Step 7:

[0937] The terminal displays automation suggestions received from the server to the user. The user reviews these suggestions and customizes them as needed.

[0938] Step 8:

[0939] The user accepts the automation suggestion or enters customized settings and sends them to the server via their device.

[0940] Step 9:

[0941] The server receives new settings sent by the user. These settings are stored in a database and used to control smart home devices.

[0942] Step 10:

[0943] Based on the new settings received by the server, it issues instructions to control smart home devices. For example, it can set lights to turn on / off or music playback time.

[0944] Step 11:

[0945] Smart home devices operate based on instructions from a server. This allows for automatic adjustment of lighting and temperature, providing an environment tailored to the user's emotional state.

[0946] Through this series of processes, the system of the present invention makes the user's life more comfortable and automates the operation of smart home devices in accordance with their emotional state.

[0947] (Example 2)

[0948] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0949] In recent years, home devices and systems have become smarter, and there is a growing demand for them to make users' lives more comfortable. However, many smart home systems only collect data on the user's living environment and perform simple automation, lacking the flexibility to consider the user's emotional state. As a result, it is difficult to provide a truly relaxing environment for the user. Furthermore, the ability for users to customize automation suggestions is insufficient. There is a need for smart home automation systems that can solve these problems and make users' lives more comfortable and pleasant.

[0950] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0951] In this invention, the server includes means for receiving data relating to the user's living environment and emotional state, means for analyzing the living environment and emotional data to generate optimal automation suggestions for the user, and means for notifying the user of the generated automation suggestions. This enables flexible automation suggestions that take into account the user's emotional state, and allows the user to customize and accept the suggestions. This makes it possible to provide the user with a more comfortable living environment that meets their individual needs.

[0952] "Living environment data" refers to information about the environment in which users live their daily lives, including data such as temperature, humidity, lighting conditions, and sound environment of the living space.

[0953] "Emotional data" refers to information that indicates a user's emotional state, and is extracted from data such as facial expressions, voice tone, and behavioral patterns.

[0954] "Automated suggestions" refer to specific operational instructions generated by the server as a result of analyzing the user's living environment data and emotional data, in order to optimize the user's living environment.

[0955] A "smart home device" is an internet-connected device used to control electrical appliances and equipment within a home, and includes lighting, air conditioning, music players, and other similar devices.

[0956] A "machine learning algorithm" is a mathematical method for learning patterns from large amounts of data and making predictions and classifications about the future.

[0957] An "emotion analysis engine" is software or an algorithm that analyzes collected emotional data to estimate a user's emotional state.

[0958] "Notification methods" refer to features that inform users of automation suggestions, and may include displays, voice notifications, and mobile applications.

[0959] "Customization options" refer to interfaces or functions that allow users to change or adjust suggested automation settings.

[0960] "Operation means" refers to a function that allows a server or other control device to send direct operation instructions to a smart home device.

[0961] A "database" is a system used by a server to store data on living environment and emotional data that it receives.

[0962] The smart home automation system of this invention makes and executes optimal automation suggestions based on the user's living environment and emotional state. This system mainly consists of a server, terminals, users, and an emotion analysis engine.

[0963] 1. Data Collection

[0964] Users use smart home devices (cameras, microphones, etc.) to record data about their daily activities and living environment. This includes video information, audio data, activity records, and living space information. For example, facial expressions and sounds while a user is watching TV in the living room may be collected.

[0965] 2. Data transmission

[0966] The device temporarily stores the collected data and then sends it to the server. Wireless communication technologies such as Wi-Fi and Bluetooth are used for this data transfer.

[0967] 3. Data storage

[0968] The server stores the received data in a cloud storage service (e.g., Amazon S3, Google Cloud Storage). The stored data includes timestamps and various metadata.

[0969] 4. Data Analysis

[0970] The server uses machine learning algorithms (e.g., TensorFlow) and sentiment analysis engines (e.g., EmotionAI) to analyze the stored data. This analysis helps understand the user's emotional state and lifestyle patterns. For example, if a user shows signs of fatigue in their facial expression or voice tone, this information is analyzed and extracted as sentiment data.

[0971] 5. Automated proposal generation

[0972] The server generates optimal automation suggestions for the user based on the analyzed data. These suggestions include specific instructions, such as setting the living room lighting to a warm color and playing relaxation music.

[0973] 6. Proposal Notification and Customization

[0974] The device notifies the user of the generated automation suggestions. These notifications are delivered via the display or voice message. The user reviews the suggestions and makes changes or customizations as needed. For example, the user can change the suggested music to jazz.

[0975] 7. Implement the proposal

[0976] The server receives suggestions reviewed and customized by the user and implements them on smart home devices. Specifically, it calls the API of a smart light bulb to change the color of the light and plays specified music through a smart speaker.

[0977] Specific example

[0978] For example, when user A returns home from work and turns on the living room lights, the emotion analysis engine detects user A's tired expression. This data is sent to the server, which generates an automated suggestion to "set the living room lights to a warm color and play relaxation music." This suggestion is displayed on the device, and if user A accepts it, the server sends instructions to the smart home device, and the lighting and music are automatically adjusted.

[0979] Example of a prompt

[0980] "In a smart home system, please explain how the system adjusts the environment when the emotion engine detects that the user is tired after returning home from work."

[0981] Thus, the system of the present invention can make the user's life more comfortable and efficient by considering the user's living environment and emotional state, and by generating and executing optimal automation suggestions.

[0982] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0983] Step 1:

[0984] Users record data about their daily activities and living environment using smart home devices (cameras, microphones, etc.). Specifically, a smart camera records the user's facial expressions in real time, and a smart microphone captures their voice tone. The input data consists of video and audio information, which is temporarily stored by the smart home devices.

[0985] Step 2:

[0986] The device receives temporarily stored data (video and audio data), compresses the data, and sends it to the server. Specifically, the device uploads the data to the cloud server using Wi-Fi. The input is data acquired from the smart home device, and the output is compressed data packets.

[0987] Step 3:

[0988] The server saves the received data to cloud storage (e.g., Amazon S3 or Google Cloud Storage). Specifically, the server adds a timestamp to the received data and transfers it to the specified storage bucket. The input data is the data packets sent from the terminal, and the output data is the data file stored in the cloud.

[0989] Step 4:

[0990] The server runs machine learning algorithms (e.g., TensorFlow) and an emotion analysis engine to analyze the stored data. Specifically, the server retrieves data from the database and inputs it into the machine learning model to analyze the user's emotional state. The analysis results include the user's emotional state, such as whether they are tired or relaxed. The input data consists of video and audio data stored in the cloud, and the output is the analyzed emotion data.

[0991] Step 5:

[0992] The server generates automated suggestions based on emotional data. Specifically, the server selects the most suitable suggestion from a template based on the analysis results and customizes it. For example, for a tired user, it might generate a suggestion to "change the lighting to a warm color and play relaxation music." The input data is the analyzed emotional data, and the output is the generated automated suggestion.

[0993] Step 6:

[0994] The terminal notifies the user of automation suggestions received from the server. Specifically, the terminal displays a pop-up notification on its screen, showing the suggestion content. The input consists of suggestion data sent from the server, and the output is a notification visible to the user.

[0995] Step 7:

[0996] The user reviews, selects, and customizes the automated suggestions they receive. Specifically, the user interacts with the terminal's interface to accept or customize suggestions. For example, they can change the relaxation music to jazz. The input data is the suggestions displayed on the terminal, and the output data is the customized suggestions.

[0997] Step 8:

[0998] The server receives customized suggestions and implements them on smart home devices. Specifically, the server calls the smart light bulb's API to change the light color and controls the smart speaker to play the specified music. The input data is the user's customized suggestions, and the output data is the settings changes made to the executed devices.

[0999] (Application Example 2)

[1000] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1001] Conventional smart home automation systems are limited to automation suggestions based on simple living environment data, making it difficult to optimize the environment while considering the emotional state of users and customers. Furthermore, in physical stores, there is a lack of means to improve the shopping experience using customer emotional data, making it challenging to improve the quality of customer service. Additionally, product recommendations and environmental adjustments are often performed manually by staff, highlighting the need for a system that can effectively assist customers.

[1002] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1003] In this invention, the server includes means for receiving data on the user's living environment and emotional data; means for analyzing the living environment data and emotional data to generate optimal automation suggestions for the user; means for notifying the user of the generated automation suggestions; means for receiving settings changed by the user based on the notified automation suggestions; means for controlling smart environment devices to execute the received settings; means for detecting customer emotional data (facial expressions, voice tone, behavior) within a physical store; and means for analyzing the emotional data and notifying store staff of specific product suggestions or environmental adjustments. This makes it possible to optimize the living environment and shopping experience based on the emotional state of the user and customers.

[1004] "Living environment data" refers to all environmental information related to the user's life, including visual sensor information, audio sensor information, activity records, and information on the layout of the residence.

[1005] "Emotional data" refers to information about the emotional state of users and customers, extracted from their facial expressions, voice tone, behavior, and other factors.

[1006] "Automated suggestions" refer to suggestions for optimal environmental settings and actions provided to users and customers, based on analysis of living environment data and emotional data.

[1007] "Smart environment devices" refer primarily to electronic devices and equipment used to automatically control the environment of homes and stores, and include lighting, air conditioning, and music playback devices.

[1008] An "emotion analysis engine" refers to an analysis system that analyzes emotional data such as facial expressions, voice tone, and behavior of users and customers in real time to understand their emotional state.

[1009] A "physical store" refers to a commercial facility that customers can physically visit, where goods are displayed and sold.

[1010] "Specific product recommendations" refer to suggesting the most suitable products and services to individual customers based on analyzed emotional data.

[1011] "Environmental adjustment" refers to automatically setting environmental elements such as lighting, music, and temperature to their optimal state based on user and customer sentiment data.

[1012] "Analysis means" refers to a combination of hardware and software for processing and analyzing received data, particularly those using machine learning algorithms or sentiment analysis engines.

[1013] "Notification means" refers to methods and devices for communicating generated automation suggestions and information to users and store staff, and includes interfaces such as smartphones and smart glasses.

[1014] System Overview

[1015] The system implementing this invention receives and analyzes user and store customer living environment data and emotional data, and generates optimal automation suggestions based on this data. The system consists of a server, terminals, users, an emotional analysis engine, and smart environment devices.

[1016] Users and customers

[1017] Users and customers collect environmental and emotional data using cameras, microphones, and other devices. This includes visual sensor information (facial expressions), voice sensor information (voice tone), and behavioral records. The collected data is transmitted to the server via the user's device.

[1018] terminal

[1019] The terminal receives user and customer data and transmits it to the server. It also has an interface for presenting automation suggestions received from the server to the user. This interface consists of devices such as smartphones and smart glasses.

[1020] server

[1021] The server is the central hub for analyzing living environment data and emotional data received from users and customers. Machine learning algorithms and an emotional analysis engine are used for the analysis. Based on the analysis results, optimal automation suggestions are generated for users and customers and notified via their devices. If a user accepts a suggestion, the settings are reflected in the smart environment device, and the environment is automatically adjusted.

[1022] Emotion analysis engine

[1023] The emotion analysis engine analyzes user and customer facial expressions, voice tone, and behavior in real time to acquire emotional data. This data is sent to a server for further analysis and used to generate automated suggestions.

[1024] Hardware and software to be used

[1025] Camera: As a visual sensor, we will use the Logitech HD Pro Webcam C920 as an example.

[1026] Microphone: As an example, we will use the Blue Yeti USB Microphone as the voice sensor.

[1027] Smartphones / smart glasses: Used as a means of notification and data collection.

[1028] Analysis software: Machine learning algorithms and sentiment analysis engines.

[1029] Specific example

[1030] For example, if a customer in a physical store is looking at a product but shows no interest (the emotion analysis engine determines this to be "lack of interest"), the server analyzes this data and notifies the staff member wearing smart glasses to "introduce the new product to customer X." This notification allows the store staff to make the most appropriate suggestions to the customer.

[1031] Example of a prompt

[1032] "emotion_data: Customer X has a dissatisfied expression, their voice tone is low, and they are not showing interest in the product."

[1033] The above describes the embodiments for carrying out this invention. This system enables optimal environmental adjustment and product recommendations based on the emotional state of users and customers.

[1034] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1035] Step 1:

[1036] Data collection from users and customers

[1037] The user collects environmental data (visual sensor information, audio sensor information, behavioral records) and emotional data using a camera and microphone. This data is transmitted to the user's device (smartphone or smart glasses).

[1038] Input: Visual sensor information, audio sensor information, and activity records collected by the camera and microphone.

[1039] Output: Living environment data and emotional data transmitted to the terminal.

[1040] Step 2:

[1041] Sending data to the server

[1042] The device sends the collected data to the server. The server receives this data and stores it in a database.

[1043] Input: Living environment data and emotional data transmitted from the device.

[1044] Output: Living environment data and emotional data stored on the server

[1045] Step 3:

[1046] Data Analysis

[1047] The server analyzes the received data using machine learning algorithms and sentiment analysis engines. This identifies the emotional state and lifestyle patterns of users and customers.

[1048] Input: Living environment data and emotional data stored on the server

[1049] Output: Analyzed emotional states and lifestyle patterns

[1050] Step 4:

[1051] Generation of automation proposals

[1052] Based on the analyzed data, the server generates optimal automation suggestions for users and customers. This includes specific product recommendations and environment adjustments.

[1053] Input: Analyzed emotional states and lifestyle patterns

[1054] Output: Generated automation proposals

[1055] Step 5:

[1056] Automation suggestion notification

[1057] The server notifies the user's device of the generated automation suggestions. The device then displays the suggestions to the user and store staff via smartphones or smart glasses.

[1058] Input: Generated automation proposals

[1059] Output: Automation suggestions displayed on the terminal

[1060] Step 6:

[1061] Received user settings change

[1062] Users and store staff change their settings based on the automated suggestions they receive. The terminal then sends these setting changes to the server.

[1063] Input: Settings changed by users and store staff

[1064] Output: Configuration changes sent to the server

[1065] Step 7:

[1066] Smart Environment Device Control

[1067] The server controls smart environment devices based on the received configuration changes. This ensures that lighting, air conditioning, music playback devices, and other devices are set to their optimal state.

[1068] Input: Configuration changes sent to the server

[1069] Output: Smart environment devices with changed settings reflected.

[1070] The above outlines the specific processing steps of this system. This makes it possible to optimize the living environment and shopping experience based on the emotional state of users and customers.

[1071] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1072] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1073] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1074] [Fourth Embodiment]

[1075] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1076] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1077] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1078] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1079] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1080] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1081] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1082] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1083] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1084] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1085] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1086] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1087] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1088] System Overview

[1089] The smart home automation system of this invention improves the user's daily life by receiving and analyzing data about the user's living environment and generating optimal automation suggestions. The system is broadly composed of three entities: a server, a terminal, and a user, which work together in cooperation.

[1090] User

[1091] Users record daily activity data using smart home devices. This includes things like turning lights on / off, changing temperature settings, and recording activity with a camera, and this data is periodically sent to a server via the device.

[1092] terminal

[1093] The terminal is responsible for receiving data sent from the user and transmitting it to the server. It also has a display and interface to present automation suggestions received from the server to the user. The user can review, select, and customize the provided automation suggestions, and these selections are also transmitted to the server via the terminal.

[1094] server

[1095] The server acts as the central hub for analyzing living environment data received from users and generating automation suggestions. The received data is stored in a database and analyzed using machine learning algorithms. This allows the system to understand the user's lifestyle patterns and needs, and generate optimal automation suggestions based on this information. The generated suggestions are then notified to the user via their device. The new settings, once accepted by the user, are then returned to the server and implemented by the smart home device's control system.

[1096] Examples of automation proposals

[1097] For example, if user A wakes up at 6 AM and turns on the living room lights, the server analyzes this pattern and generates an automation suggestion such as "Automatically turn on the living room lights at 6 AM." It also suggests "Automatically turn off the living room lights" at 8 AM when user A leaves for work. This allows user A to automate simple daily tasks and live a more efficient and comfortable life.

[1098] Other features

[1099] Furthermore, the system allows for user-specific customization. For example, it supports changing settings only on specific days, and automating settings based on advanced conditions (temperature, day of the week, etc.). Users input these custom settings from their devices, and the server retrieves those settings and controls the operation of smart home devices.

[1100] Examples

[1101] As a concrete example, consider a case where user B wants to set up their air conditioner to automatically adjust its temperature according to their working hours. The system receives working hour data for each day of the week and makes a suggestion to optimize the air conditioner temperature based on that data. If user B accepts the suggestion, the server applies the setting to the smart home device, and the air conditioner temperature is automatically adjusted.

[1102] Thus, the present invention is a powerful tool for simplifying users' daily lives and improving their quality of life.

[1103] The following describes the processing flow.

[1104] Step 1:

[1105] Users record daily activity data using smart home devices. This data includes turning lights on / off, changing temperature settings, and recording activity with cameras.

[1106] Step 2:

[1107] The device receives activity data recorded by the user and periodically sends it to the server. This transmission takes place over the internet.

[1108] Step 3:

[1109] The server receives living environment data sent from the terminal. The received data is stored in a database.

[1110] Step 4:

[1111] The server analyzes the received living environment data and runs machine learning algorithms to understand the user's lifestyle patterns and needs. This analysis identifies the user's habits and specific needs.

[1112] Step 5:

[1113] The server generates automation suggestions based on the analysis results. For example, if a user has a habit of waking up at 6 AM and turning on the living room lights, the server will generate a suggestion to "automatically turn on the living room lights at 6 AM."

[1114] Step 6:

[1115] The server sends the automation suggestions it generates to the terminal.

[1116] Step 7:

[1117] The terminal displays automation suggestions received from the server to the user. The user reviews these suggestions and customizes them as needed.

[1118] Step 8:

[1119] The user accepts the automation suggestion or enters customized settings and sends them to the server via their device.

[1120] Step 9:

[1121] The server receives new settings sent by the user. These settings are stored in a database and used to control smart home devices.

[1122] Step 10:

[1123] Based on the new settings received by the server, it issues instructions to control smart home devices. For example, it can set the on / off times for lights.

[1124] Step 11:

[1125] Smart home devices operate based on instructions from a server. This automates functions such as lighting and temperature control.

[1126] Through this series of processes, the present invention makes the user's life more comfortable and simplifies the operation of smart home devices.

[1127] (Example 1)

[1128] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1129] Traditional smart home automation systems required users to manually configure individual devices, and did not offer optimal suggestions based on lifestyle patterns or needs. This resulted in increased user effort and frequency of operation, ultimately diminishing convenience.

[1130] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1131] In this invention, the server includes means for collecting data about the user's living environment, means for periodically transmitting the living environment data to the server, and means for storing and analyzing the received living environment data in a database. This makes it possible to generate optimal automation suggestions based on the user's lifestyle patterns and needs, and notify the user in a form that can be easily customized. This reduces the effort required from the user and provides a more comfortable and efficient smart home experience.

[1132] A "user" is someone who uses a smart home system, provides data about their living environment, and utilizes the automation suggestions provided by the system.

[1133] "Living environment data" refers to information collected through smart home devices, such as the user's indoor activities and device settings.

[1134] A "server" is a computer system that stores and analyzes received data, generates automation suggestions, and notifies the user.

[1135] A "database" is a place on a server where user-generated data about their living environment is organized, stored, and used for later analysis.

[1136] "Analyzing" means using data analysis algorithms based on collected living environment data to understand the user's lifestyle patterns and needs.

[1137] "Automation suggestions" are information that indicates setting changes or new operating methods to make the user's life more convenient, based on the results of an analysis of living environment data.

[1138] "Notifying" refers to the act of informing the user of the generated automation suggestions, and this is done through the notification function of a smartphone or the interface of a dedicated application.

[1139] "Customization" refers to the process by which users can modify and adjust automated suggestions they receive to suit their own needs.

[1140] "Smart home devices" refer to hardware devices such as lighting, air conditioners, and cameras in the home that automate their operation based on control instructions from a server.

[1141] A "data analysis algorithm" is a computational method used to analyze collected data, employing machine learning techniques and statistical analysis techniques.

[1142] This invention is a smart home system that collects and analyzes data related to the user's living environment and generates optimal automation suggestions based on the results. This system operates through the mutual cooperation of three entities: the user, the terminal, and the server.

[1143] User

[1144] Users are the entities that provide data about their living environment. Users use smart home devices to record daily activity data. This includes things like turning living room lights on / off, changing the air conditioner temperature, and recording activities with a camera. This data is periodically sent to the server via the device.

[1145] terminal

[1146] The terminal is responsible for receiving data sent from the user and transmitting it to the server. It also features a display or interface (e.g., a smartphone app) to present automation suggestions received from the server to the user. The user can review the provided automation suggestions and customize them as needed. This customization information is also transmitted to the server via the terminal.

[1147] server

[1148] The server has a central function of analyzing living environment data received from the user and generating automation suggestions. The server stores the received data in a database (e.g., MySQL or PostgreSQL). Next, the stored data is analyzed using machine learning algorithms with analysis software such as Python or R. This allows the server to understand the user's living patterns and needs. Based on this, optimal automation suggestions are generated and notified to the user via their device. If the user accepts the suggestion, the new settings are sent back to the server and reflected in the control of the smart home devices. IoT protocols such as MQTT and CoAP are used for control.

[1149] Specific example

[1150] For example, if user A wakes up at 6 AM and turns on the living room lights, the server analyzes this pattern and generates an automation suggestion to "automatically turn on the living room lights at 6 AM." It also suggests "automatically turn off the living room lights" at 8 AM when user A leaves for work. This allows user A to automate simple daily tasks, making their life more convenient.

[1151] Example of a prompt

[1152] An example of a prompt message for inputting a specific example into a generative AI model is as follows:

[1153] User B wants the air conditioner temperature to be automatically adjusted according to their working hours. We want to receive working hour data for each day of the week and then suggest an optimized temperature based on that data. Please explain in detail how this system works.

[1154] The above describes the modes for carrying out the invention. This invention is designed with the aim of making the user's life more efficient and improving their comfort.

[1155] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1156] Step 1: Data Collection

[1157] Users use smart home devices to collect daily activity data. This data includes turning lights on / off, changing air conditioner temperature settings, and recording information from cameras. For example, a user turns on the living room lights. This action changes the light status to "on," and this data is recorded on the device. The input is the user's action, and the output is the log data of that action.

[1158] Step 2: Data transmission

[1159] The terminal periodically sends collected data to the server. It uses communication methods such as Wi-Fi or Bluetooth to generate data packets and transmit them via a secure protocol (such as HTTPS). For example, the terminal sends the user's air conditioner settings data to the server. The input is the operation data stored on the terminal, and the output is the data packets sent to the server.

[1160] Step 3: Save Data

[1161] The server stores the received data in a database. The database used is a database management system such as MySQL or PostgreSQL. For example, the server stores air conditioner setting data received from a terminal in the database. The input is the received data packet, and the output is the data recorded in the database.

[1162] Step 4: Data Analysis

[1163] The server uses stored data to execute data analysis algorithms (using Python or R) and analyze the user's lifestyle patterns. Machine learning techniques are used to extract patterns and trends from the data. For example, the server analyzes the user's lighting operation data to detect a pattern where the living room lights are turned on at 6 AM every morning. The input is lifestyle environment data stored in a database, and the output is lifestyle pattern data obtained through analysis.

[1164] Step 5: Generating automation proposals

[1165] Based on the analysis results, the server generates optimal automation suggestions. For example, the server might generate a suggestion to "automatically turn on the living room lights at 6 AM." The generated suggestions are then sent to the terminal. The input is lifestyle pattern data, and the output is automation suggestions.

[1166] Step 6: User presentation of the proposal

[1167] The terminal presents the user with automation suggestions received from the server. This is done through the smartphone's notification function or a dedicated application interface. The user reviews the suggestions and customizes them as needed. For example, the terminal notifies the user of a suggestion to "turn on the living room lights at 6 AM," and the user reviews and accepts the suggestion. The input is the automation suggestion sent from the server, and the output is the user's customized data.

[1168] Step 7: Submit customization information

[1169] After the user customizes the suggestion, the device sends that information to the server. For example, if the user changes the suggestion to "Turn on the living room lights at 7 AM instead of 6 AM," that customization information will be sent. The input is the user's customization data, and the output is the configuration change data sent to the server.

[1170] Step 8: Apply the settings

[1171] The server controls smart home devices based on the received customization information. It uses IoT protocols (such as MQTT or CoAP) to apply the new settings to the devices. For example, the server automatically adjusts the air conditioner temperature based on the user's working hours. The input is the retransmitted configuration change data, and the output is the applied device settings.

[1172] Through the steps described above, optimal automation based on the user's lifestyle patterns is achieved.

[1173] (Application Example 1)

[1174] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1175] Traditional smart home automation systems primarily aimed to streamline daily life by generating optimal automation suggestions based on data about the user's living environment. However, applying this technology to physical stores has great potential to improve store operations and enhance customer satisfaction. For example, there is a need to maintain comfort within stores by automatically adjusting lighting and air conditioning, to increase purchasing intent by providing appropriate promotions based on customer behavior patterns, and to strengthen security within stores. However, such a comprehensive store management system does not yet exist, and its realization remains a challenge.

[1176] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1177] In this invention, the server includes means for receiving data relating to the user's living environment, means for analyzing the living environment data to generate optimal automation suggestions for the user, means for notifying the user of the generated automation suggestions, means for receiving settings changed by the user based on the notified automation suggestions, means for controlling an operating device for executing the received settings, means for automatically adjusting lighting and air conditioning settings based on store environment data, means for sending promotional notifications based on the customer's surroundings, and means for monitoring in-store security, detecting anomalies, and providing notifications. This makes it possible to streamline store operations, improve customer satisfaction, and provide a secure environment.

[1178] "Data related to the user's living environment" refers to all information related to the user's daily life, and specifically includes lighting usage, temperature settings, activity records, etc.

[1179] An "automation proposal" is a specific suggestion that presents the user with the most suitable automation method based on the received data.

[1180] "Means of notification" refers to communication methods or interfaces used to inform users of generated automation suggestions.

[1181] "Operating devices" refer to various devices and equipment used to execute automation settings modified by the user. Examples include smart home devices and in-store equipment.

[1182] "Store environment data" refers to information related to the conditions inside the store, including lighting brightness, foot traffic, and temperature.

[1183] "Means of sending promotional notifications based on customer location" refers to a function that sends promotional content in real time based on the customer's location information and behavioral patterns.

[1184] "Means of monitoring in-store security and detecting and notifying of anomalies" refers to a system that uses surveillance cameras and other sensors to monitor the safety of the store and issues an alert when suspicious behavior or anomalies are detected.

[1185] "Video information" refers to video stream data obtained from cameras and other video acquisition devices.

[1186] "Behavioral records" refer to information that records the movements and behavioral patterns of users and customers.

[1187] "Store layout information" refers to detailed information about the layout and arrangement of items within a store.

[1188] A "machine learning algorithm" refers to an algorithm that analyzes data, automatically learns patterns and trends, and uses that information to inform future suggestions and control measures.

[1189] This invention aims to improve the efficiency of store operations and enhance customer satisfaction by utilizing data related to the user's living environment. Specific embodiments of this invention will be described in detail below.

[1190] System Overview

[1191] The smart physical store management system of the present invention is a system composed of three components: a server, a terminal, and a user, which work together in cooperation with each other.

[1192] User roles

[1193] The user collects environmental data (temperature, lighting, foot traffic, etc.) from the store using various sensors and devices installed within the store. These devices include temperature sensors, lighting control systems, and surveillance cameras. Sensor information is periodically transmitted to a terminal.

[1194] Terminal role

[1195] The terminal receives data sent by the user and forwards it to the server. Simultaneously, it has an interface for displaying and presenting automated suggestions and notifications received from the server to the user. Through this interface, the user can review, select, and customize the suggestions.

[1196] Server Role

[1197] The server is a central device that analyzes living environment data received from users and generates optimal automation suggestions. Specifically, it performs the following processes:

[1198] (1) Analysis of received data

[1199] Received data is stored in a database and analyzed using machine learning algorithms. For example, it learns lighting usage patterns and air conditioning settings to generate optimal automation suggestions.

[1200] (2) Generation of automation proposals

[1201] Based on the analyzed data, the system generates automation suggestions. For example, it might suggest increasing the lighting and lowering the air conditioning temperature during peak hours in a store.

[1202] (3) Notification and execution

[1203] The generated automation suggestions are notified to the user via their device. If the user accepts the suggestion, the settings are returned to the server and applied to the various devices.

[1204] Other features

[1205] Automatic lighting and air conditioning management: Lighting and air conditioning settings are automatically adjusted based on foot traffic and time of day within the store.

[1206] Promotional notifications: Send promotional information and discount coupons via push notifications based on specific conditions (e.g., when the customer's device is in a specific area).

[1207] Inventory Management Assist: Analyzes inventory information in real time and suggests adding out-of-stock items.

[1208] Security monitoring: Analyzes footage from in-store cameras to automatically detect and notify of suspicious activity.

[1209] Hardware and software to be used

[1210] Hardware: Temperature sensors, lighting control systems, surveillance cameras, user interface terminals

[1211] Software: AWS IoT Core, database system, machine learning algorithms, push notification API

[1212] Specific example

[1213] For example, at 5 PM when the store is crowded, the system automatically brightens the lights and lowers the air conditioning temperature to provide a comfortable environment. Also, if there are new promotional items, customers in specific areas will receive real-time promotional notifications such as "20% off new items!"

[1214] Example input prompts for a generative AI model

[1215] Please create a promotional notification feature for your smart physical store management system based on the following conditions:

[1216] Send push notifications to customer devices when certain conditions are met.

[1217] The system analyzes sensor data within the store and sends notifications at the optimal time.

[1218] Use Python to generate code that integrates with AWS IoT Core and its API.

[1219] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1220] Step 1:

[1221] The user receives store environment data acquired from multiple sensors (temperature sensors, lighting control systems, surveillance cameras, etc.). This includes temperature, foot traffic, lighting brightness, and video. The user sends this data to their terminal.

[1222] Inputs: Temperature, lighting, pedestrian flow, surveillance camera data

[1223] Output: Store environment data is sent to the terminal.

[1224] Step 2:

[1225] The terminal receives environmental data sent by the user and formats it for transfer to the server. The formatted data is sent to the server in real time.

[1226] Input: Store environment data submitted by the user

[1227] Output: Formatted environment data is sent to the server.

[1228] Step 3:

[1229] The server stores the received environmental data in a database and analyzes the data using machine learning algorithms. For example, it learns lighting usage patterns, air conditioning settings, and customer behavior patterns.

[1230] Input: Formatted store environment data

[1231] Output: The learning model is saved in the database as an analysis result.

[1232] Step 4:

[1233] The server generates optimal automation suggestions based on the analysis results. For example, it might suggest increasing the lighting and lowering the air conditioning temperature during busy hours.

[1234] Input: Analysis results

[1235] Output: Specific automation proposals

[1236] Step 5:

[1237] The server notifies the terminal of the generated automation proposal. The terminal receives this proposal and notifies the user through the interface.

[1238] Input: Automation proposal

[1239] Output: Suggestions notified to the terminal

[1240] Step 6:

[1241] The user reviews the automated suggestions notified via their device, changes the settings as needed, and then returns them to the server. For example, they might customize the air conditioner temperature setting from 22 degrees to 24 degrees.

[1242] Input: Automation suggestions from the terminal

[1243] Output: User-modified settings

[1244] Step 7:

[1245] The server receives the settings returned by the user and applies them to the various operating devices. Specifically, it controls smart home devices to apply the new settings.

[1246] Input: User-modified settings

[1247] Output: New settings reflected in the operating device

[1248] Step 8:

[1249] The server monitors processing results and checks security within the store, issuing alerts if it detects any unusual activity. For example, if a security camera captures suspicious behavior, it will notify the user in real time.

[1250] Input: Security data within the store

[1251] Output: Alert notification

[1252] Step 9:

[1253] The server generates promotional notifications based on the customer's location and behavioral patterns, and sends push notifications to the customer via their device.

[1254] Input: Customer location information and behavioral patterns

[1255] Output: Promotional notification sent to customer

[1256] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1257] System Overview

[1258] The smart home automation system of this invention receives and analyzes user living environment data and emotional data, and provides the user with optimal automation suggestions. This system consists of a server, a terminal, a user, and an emotional engine.

[1259] User

[1260] Users record daily activity and emotional data through smart home devices and an emotion engine. Emotional data is extracted from the user's facial expressions, voice tone, and behavior using cameras and microphones. This data is transmitted to a server via the device.

[1261] terminal

[1262] The terminal receives lifestyle and emotional data sent by the user and transmits it to the server. It also has a display and interface to present automation suggestions received from the server to the user. The user can review, select, and customize the provided automation suggestions, and these selections are also transmitted to the server via the terminal.

[1263] server

[1264] The server acts as the central hub for analyzing lifestyle and emotional data received from users and generating automation suggestions. The received data is stored in a database and analyzed using machine learning algorithms and an emotional analysis engine. This allows the system to understand the user's lifestyle patterns and emotional state, and generate optimal automation suggestions based on this information. The generated suggestions are then notified to the user via their device. The new settings, once accepted by the user, are then returned to the server and implemented by the smart home device's control system.

[1265] Emotional Engine

[1266] The emotion engine analyzes the user's facial expressions, voice tone, and behavior in real time to acquire emotional data. This allows the system to understand the user's emotional state, and this information is then sent to the server.

[1267] Examples of automation proposals

[1268] For example, if user A returns home from work and turns on the living room lights, and their facial expression is detected as tired (by the emotion engine), the server analyzes this data and generates an automated suggestion such as "set the living room lights to a warm color and play relaxation music." This suggestion automatically provides user A with a comfortable environment.

[1269] Other features

[1270] Furthermore, the system allows for customization based on the user's emotions. For example, it supports features such as setting the temperature slightly lower if the user is feeling stressed, or automation based on advanced conditions (temperature, day of the week, etc.). Users enter these customization settings from their device, and the server takes those settings and controls the operation of smart home devices.

[1271] Examples

[1272] For example, if the emotion engine detects that user B is spending more time in the living room and is experiencing stress, the server uses this data to generate a suggestion to change the lighting to a softer light and set the temperature to a comfortable level. If user B accepts the suggestion, the server applies those settings to the smart home devices and automatically adjusts the environment.

[1273] Thus, the present invention makes users' lives more comfortable and efficient, and realizes flexible automation that adapts to their emotional state.

[1274] The following describes the processing flow.

[1275] Step 1:

[1276] Users record daily activity and emotional data using smart home devices and an emotion engine. Activity data includes turning lights on / off and changing temperature settings, while emotional data includes facial expressions and voice tones captured through cameras and microphones.

[1277] Step 2:

[1278] The device receives activity and emotion data recorded by the user and periodically sends it to a server. This transmission takes place over the internet.

[1279] Step 3:

[1280] The server receives living environment data and emotional data sent from the terminal. The received data is stored in a database.

[1281] Step 4:

[1282] The server analyzes the received living environment data and emotional data. Machine learning algorithms and an emotional analysis engine are used for the analysis to identify the user's lifestyle and emotional state.

[1283] Step 5:

[1284] The server generates automation suggestions based on the analysis results. For example, it might generate a suggestion such as, "If the user is fatigued at night, change the lighting to a warm color and play relaxation music."

[1285] Step 6:

[1286] The server sends the automation suggestions it generates to the terminal.

[1287] Step 7:

[1288] The terminal displays automation suggestions received from the server to the user. The user reviews these suggestions and customizes them as needed.

[1289] Step 8:

[1290] The user accepts the automation suggestion or enters customized settings and sends them to the server via their device.

[1291] Step 9:

[1292] The server receives new settings sent by the user. These settings are stored in a database and used to control smart home devices.

[1293] Step 10:

[1294] Based on the new settings received by the server, it issues instructions to control smart home devices. For example, it can set lights to turn on / off or music playback time.

[1295] Step 11:

[1296] Smart home devices operate based on instructions from a server. This allows for automatic adjustment of lighting and temperature, providing an environment tailored to the user's emotional state.

[1297] Through this series of processes, the system of the present invention makes the user's life more comfortable and automates the operation of smart home devices in accordance with their emotional state.

[1298] (Example 2)

[1299] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1300] In recent years, home devices and systems have become smarter, and there is a growing demand for them to make users' lives more comfortable. However, many smart home systems only collect data on the user's living environment and perform simple automation, lacking the flexibility to consider the user's emotional state. As a result, it is difficult to provide a truly relaxing environment for the user. Furthermore, the ability for users to customize automation suggestions is insufficient. There is a need for smart home automation systems that can solve these problems and make users' lives more comfortable and pleasant.

[1301] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1302] In this invention, the server includes means for receiving data relating to the user's living environment and emotional state, means for analyzing the living environment and emotional data to generate optimal automation suggestions for the user, and means for notifying the user of the generated automation suggestions. This enables flexible automation suggestions that take into account the user's emotional state, and allows the user to customize and accept the suggestions. This makes it possible to provide the user with a more comfortable living environment that meets their individual needs.

[1303] "Living environment data" refers to information about the environment in which users live their daily lives, including data such as temperature, humidity, lighting conditions, and sound environment of the living space.

[1304] "Emotional data" refers to information that indicates a user's emotional state, and is extracted from data such as facial expressions, voice tone, and behavioral patterns.

[1305] "Automated suggestions" refer to specific operational instructions generated by the server as a result of analyzing the user's living environment data and emotional data, in order to optimize the user's living environment.

[1306] A "smart home device" is an internet-connected device used to control electrical appliances and equipment within a home, and includes lighting, air conditioning, music players, and other similar devices.

[1307] A "machine learning algorithm" is a mathematical method for learning patterns from large amounts of data and making predictions and classifications about the future.

[1308] An "emotion analysis engine" is software or an algorithm that analyzes collected emotional data to estimate a user's emotional state.

[1309] "Notification methods" refer to features that inform users of automation suggestions, and may include displays, voice notifications, and mobile applications.

[1310] "Customization options" refer to interfaces or functions that allow users to change or adjust suggested automation settings.

[1311] "Operation means" refers to a function that allows a server or other control device to send direct operation instructions to a smart home device.

[1312] A "database" is a system used by a server to store data on living environment and emotional data that it receives.

[1313] The smart home automation system of this invention makes and executes optimal automation suggestions based on the user's living environment and emotional state. This system mainly consists of a server, terminals, users, and an emotion analysis engine.

[1314] 1. Data Collection

[1315] Users use smart home devices (cameras, microphones, etc.) to record data about their daily activities and living environment. This includes video information, audio data, activity records, and living space information. For example, facial expressions and sounds while a user is watching TV in the living room may be collected.

[1316] 2. Data transmission

[1317] The device temporarily stores the collected data and then sends it to the server. Wireless communication technologies such as Wi-Fi and Bluetooth are used for this data transfer.

[1318] 3. Data storage

[1319] The server stores the received data in a cloud storage service (e.g., Amazon S3, Google Cloud Storage). The stored data includes timestamps and various metadata.

[1320] 4. Data Analysis

[1321] The server uses machine learning algorithms (e.g., TensorFlow) and sentiment analysis engines (e.g., EmotionAI) to analyze the stored data. This analysis helps understand the user's emotional state and lifestyle patterns. For example, if a user shows signs of fatigue in their facial expression or voice tone, this information is analyzed and extracted as sentiment data.

[1322] 5. Automated proposal generation

[1323] The server generates optimal automation suggestions for the user based on the analyzed data. These suggestions include specific instructions, such as setting the living room lighting to a warm color and playing relaxation music.

[1324] 6. Proposal Notification and Customization

[1325] The device notifies the user of the generated automation suggestions. These notifications are delivered via the display or voice message. The user reviews the suggestions and makes changes or customizations as needed. For example, the user can change the suggested music to jazz.

[1326] 7. Implement the proposal

[1327] The server receives suggestions reviewed and customized by the user and implements them on smart home devices. Specifically, it calls the API of a smart light bulb to change the color of the light and plays specified music through a smart speaker.

[1328] Specific example

[1329] For example, when user A returns home from work and turns on the living room lights, the emotion analysis engine detects user A's tired expression. This data is sent to the server, which generates an automated suggestion to "set the living room lights to a warm color and play relaxation music." This suggestion is displayed on the device, and if user A accepts it, the server sends instructions to the smart home device, and the lighting and music are automatically adjusted.

[1330] Example of a prompt

[1331] "In a smart home system, please explain how the system adjusts the environment when the emotion engine detects that the user is tired after returning home from work."

[1332] Thus, the system of the present invention can make the user's life more comfortable and efficient by considering the user's living environment and emotional state, and by generating and executing optimal automation suggestions.

[1333] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1334] Step 1:

[1335] Users record data about their daily activities and living environment using smart home devices (cameras, microphones, etc.). Specifically, a smart camera records the user's facial expressions in real time, and a smart microphone captures their voice tone. The input data consists of video and audio information, which is temporarily stored by the smart home devices.

[1336] Step 2:

[1337] The device receives temporarily stored data (video and audio data), compresses the data, and sends it to the server. Specifically, the device uploads the data to the cloud server using Wi-Fi. The input is data acquired from the smart home device, and the output is compressed data packets.

[1338] Step 3:

[1339] The server saves the received data to cloud storage (e.g., Amazon S3 or Google Cloud Storage). Specifically, the server adds a timestamp to the received data and transfers it to the specified storage bucket. The input data is the data packets sent from the terminal, and the output data is the data file stored in the cloud.

[1340] Step 4:

[1341] The server runs machine learning algorithms (e.g., TensorFlow) and an emotion analysis engine to analyze the stored data. Specifically, the server retrieves data from the database and inputs it into the machine learning model to analyze the user's emotional state. The analysis results include the user's emotional state, such as whether they are tired or relaxed. The input data consists of video and audio data stored in the cloud, and the output is the analyzed emotion data.

[1342] Step 5:

[1343] The server generates automated suggestions based on emotional data. Specifically, the server selects the most suitable suggestion from a template based on the analysis results and customizes it. For example, for a tired user, it might generate a suggestion to "change the lighting to a warm color and play relaxation music." The input data is the analyzed emotional data, and the output is the generated automated suggestion.

[1344] Step 6:

[1345] The terminal notifies the user of automation suggestions received from the server. Specifically, the terminal displays a pop-up notification on its screen, showing the suggestion content. The input consists of suggestion data sent from the server, and the output is a notification visible to the user.

[1346] Step 7:

[1347] The user reviews, selects, and customizes the automated suggestions they receive. Specifically, the user interacts with the terminal's interface to accept or customize suggestions. For example, they can change the relaxation music to jazz. The input data is the suggestions displayed on the terminal, and the output data is the customized suggestions.

[1348] Step 8:

[1349] The server receives customized suggestions and implements them on smart home devices. Specifically, the server calls the smart light bulb's API to change the light color and controls the smart speaker to play the specified music. The input data is the user's customized suggestions, and the output data is the settings changes made to the executed devices.

[1350] (Application Example 2)

[1351] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1352] Conventional smart home automation systems are limited to automation suggestions based on simple living environment data, making it difficult to optimize the environment while considering the emotional state of users and customers. Furthermore, in physical stores, there is a lack of means to improve the shopping experience using customer emotional data, making it challenging to improve the quality of customer service. Additionally, product recommendations and environmental adjustments are often performed manually by staff, highlighting the need for a system that can effectively assist customers.

[1353] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1354] In this invention, the server includes means for receiving data on the user's living environment and emotional data; means for analyzing the living environment data and emotional data to generate optimal automation suggestions for the user; means for notifying the user of the generated automation suggestions; means for receiving settings changed by the user based on the notified automation suggestions; means for controlling smart environment devices to execute the received settings; means for detecting customer emotional data (facial expressions, voice tone, behavior) within a physical store; and means for analyzing the emotional data and notifying store staff of specific product suggestions or environmental adjustments. This makes it possible to optimize the living environment and shopping experience based on the emotional state of the user and customers.

[1355] "Living environment data" refers to all environmental information related to the user's life, including visual sensor information, audio sensor information, activity records, and information on the layout of the residence.

[1356] "Emotional data" refers to information about the emotional state of users and customers, extracted from their facial expressions, voice tone, behavior, and other factors.

[1357] "Automated suggestions" refer to suggestions for optimal environmental settings and actions provided to users and customers, based on analysis of living environment data and emotional data.

[1358] "Smart environment devices" refer primarily to electronic devices and equipment used to automatically control the environment of homes and stores, and include lighting, air conditioning, and music playback devices.

[1359] An "emotion analysis engine" refers to an analysis system that analyzes emotional data such as facial expressions, voice tone, and behavior of users and customers in real time to understand their emotional state.

[1360] A "physical store" refers to a commercial facility that customers can physically visit, where goods are displayed and sold.

[1361] "Specific product recommendations" refer to suggesting the most suitable products and services to individual customers based on analyzed emotional data.

[1362] "Environmental adjustment" refers to automatically setting environmental elements such as lighting, music, and temperature to their optimal state based on user and customer sentiment data.

[1363] "Analysis means" refers to a combination of hardware and software for processing and analyzing received data, particularly those using machine learning algorithms or sentiment analysis engines.

[1364] "Notification means" refers to methods and devices for communicating generated automation suggestions and information to users and store staff, and includes interfaces such as smartphones and smart glasses.

[1365] System Overview

[1366] The system implementing this invention receives and analyzes user and store customer living environment data and emotional data, and generates optimal automation suggestions based on this data. The system consists of a server, terminals, users, an emotional analysis engine, and smart environment devices.

[1367] Users and customers

[1368] Users and customers collect environmental and emotional data using cameras, microphones, and other devices. This includes visual sensor information (facial expressions), voice sensor information (voice tone), and behavioral records. The collected data is transmitted to the server via the user's device.

[1369] terminal

[1370] The terminal receives user and customer data and transmits it to the server. It also has an interface for presenting automation suggestions received from the server to the user. This interface consists of devices such as smartphones and smart glasses.

[1371] server

[1372] The server is the central hub for analyzing living environment data and emotional data received from users and customers. Machine learning algorithms and an emotional analysis engine are used for the analysis. Based on the analysis results, optimal automation suggestions are generated for users and customers and notified via their devices. If a user accepts a suggestion, the settings are reflected in the smart environment device, and the environment is automatically adjusted.

[1373] Emotion analysis engine

[1374] The emotion analysis engine analyzes user and customer facial expressions, voice tone, and behavior in real time to acquire emotional data. This data is sent to a server for further analysis and used to generate automated suggestions.

[1375] Hardware and software to be used

[1376] Camera: As a visual sensor, we will use the Logitech HD Pro Webcam C920 as an example.

[1377] Microphone: As an example, we will use the Blue Yeti USB Microphone as the voice sensor.

[1378] Smartphones / smart glasses: Used as a means of notification and data collection.

[1379] Analysis software: Machine learning algorithms and sentiment analysis engines.

[1380] Specific example

[1381] For example, if a customer in a physical store is looking at a product but shows no interest (the emotion analysis engine determines this to be "lack of interest"), the server analyzes this data and notifies the staff member wearing smart glasses to "introduce the new product to customer X." This notification allows the store staff to make the most appropriate suggestions to the customer.

[1382] Example of a prompt

[1383] "emotion_data: Customer X has a dissatisfied expression, their voice tone is low, and they are not showing interest in the product."

[1384] The above describes the embodiments for carrying out this invention. This system enables optimal environmental adjustment and product recommendations based on the emotional state of users and customers.

[1385] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1386] Step 1:

[1387] Data collection from users and customers

[1388] The user collects environmental data (visual sensor information, audio sensor information, behavioral records) and emotional data using a camera and microphone. This data is transmitted to the user's device (smartphone or smart glasses).

[1389] Input: Visual sensor information, audio sensor information, and activity records collected by the camera and microphone.

[1390] Output: Living environment data and emotional data transmitted to the terminal.

[1391] Step 2:

[1392] Sending data to the server

[1393] The device sends the collected data to the server. The server receives this data and stores it in a database.

[1394] Input: Living environment data and emotional data transmitted from the device.

[1395] Output: Living environment data and emotional data stored on the server

[1396] Step 3:

[1397] Data Analysis

[1398] The server analyzes the received data using machine learning algorithms and sentiment analysis engines. This identifies the emotional state and lifestyle patterns of users and customers.

[1399] Input: Living environment data and emotional data stored on the server

[1400] Output: Analyzed emotional states and lifestyle patterns

[1401] Step 4:

[1402] Generation of automation proposals

[1403] Based on the analyzed data, the server generates optimal automation suggestions for users and customers. This includes specific product recommendations and environment adjustments.

[1404] Input: Analyzed emotional states and lifestyle patterns

[1405] Output: Generated automation proposals

[1406] Step 5:

[1407] Automation suggestion notification

[1408] The server notifies the user's device of the generated automation suggestions. The device then displays the suggestions to the user and store staff via smartphones or smart glasses.

[1409] Input: Generated automation proposals

[1410] Output: Automation suggestions displayed on the terminal

[1411] Step 6:

[1412] Received user settings change

[1413] Users and store staff change their settings based on the automated suggestions they receive. The terminal then sends these setting changes to the server.

[1414] Input: Settings changed by users and store staff

[1415] Output: Configuration changes sent to the server

[1416] Step 7:

[1417] Smart Environment Device Control

[1418] The server controls smart environment devices based on the received configuration changes. This ensures that lighting, air conditioning, music playback devices, and other devices are set to their optimal state.

[1419] Input: Configuration changes sent to the server

[1420] Output: Smart environment devices with changed settings reflected.

[1421] The above outlines the specific processing steps of this system. This makes it possible to optimize the living environment and shopping experience based on the emotional state of users and customers.

[1422] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1423] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1424] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1425] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1426] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1427] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1428] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1429] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1430] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1431] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1432] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1433] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1434] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1435] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1436] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1437] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1438] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1439] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1440] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1441] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1442] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1443] The following is further disclosed regarding the embodiments described above.

[1444] (Claim 1)

[1445] A means of receiving data about the user's living environment,

[1446] A means for analyzing the aforementioned living environment data to generate optimal automation suggestions for the user,

[1447] A means for notifying the user of the generated automation proposal,

[1448] A means for receiving settings changed by the user based on the aforementioned automated proposal,

[1449] Means for controlling a smart home device to execute the received settings,

[1450] A system that includes this.

[1451] (Claim 2)

[1452] The system according to claim 1, wherein the aforementioned living environment data includes camera information, activity records, and floor plan information of the residence.

[1453] (Claim 3)

[1454] The system according to claim 1, wherein the analysis means learns the user's lifestyle patterns using a machine learning algorithm.

[1455] "Example 1"

[1456] (Claim 1)

[1457] Means for collecting data about the user's living environment,

[1458] A means for periodically transmitting the aforementioned living environment data to a server,

[1459] A means of storing and analyzing received living environment data in a database,

[1460] A means for generating an optimal automation proposal based on the analyzed data,

[1461] A means of notifying the user of the generated automation proposal and presenting it in a form that the user can customize,

[1462] A means for receiving settings changed by the user based on the aforementioned automated proposal,

[1463] Means for controlling a smart home device in order to execute the received settings,

[1464] A system that includes this.

[1465] (Claim 2)

[1466] The system according to claim 1, wherein the aforementioned living environment data includes video information, activity data, and building information.

[1467] (Claim 3)

[1468] The system according to claim 1, wherein the analysis means learns the user's lifestyle patterns using a data analysis algorithm.

[1469] "Application Example 1"

[1470] (Claim 1)

[1471] A means of receiving data about the user's living environment,

[1472] A means for analyzing the aforementioned living environment data to generate optimal automation suggestions for the user,

[1473] A means for notifying the user of the generated automation proposal,

[1474] A means for receiving settings changed by the user based on the aforementioned automated proposal,

[1475] Means for controlling an operating device for executing the received settings,

[1476] A method for automatically adjusting lighting and air conditioning settings based on store environment data,

[1477] A means of sending promotional notifications based on customer location,

[1478] A means of monitoring in-store security, detecting anomalies, and providing notifications.

[1479] A system that includes this.

[1480] (Claim 2)

[1481] The system according to claim 1, wherein the aforementioned living environment data includes video information, behavioral records, and store location information.

[1482] (Claim 3)

[1483] The system according to claim 1, wherein the analysis means learns user and store activity patterns using a machine learning algorithm.

[1484] "Example 2 of combining an emotion engine"

[1485] (Claim 1)

[1486] A means of receiving data regarding the user's living environment and emotional state,

[1487] A means for analyzing the aforementioned living environment and emotional data to generate optimal automation suggestions for the user,

[1488] A means for notifying the user of the generated automation proposal,

[1489] A means for receiving settings selected and customized by the user based on the aforementioned notified automation proposal,

[1490] Means for operating a living environment control device for executing the received settings,

[1491] A system that includes this.

[1492] (Claim 2)

[1493] The system according to claim 1, wherein the aforementioned living environment and emotional data includes video information, audio data, activity records, and living space information.

[1494] (Claim 3)

[1495] The system according to claim 1, wherein the analysis means learns the user's lifestyle patterns and emotional state using a machine learning algorithm and an emotion analysis engine.

[1496] "Application example 2 when combining with an emotional engine"

[1497] (Claim 1)

[1498] A means of receiving data about the user's living environment,

[1499] A means for analyzing the aforementioned living environment data and emotional data to generate optimal automation suggestions for the user,

[1500] A means for notifying the user of the generated automation proposal,

[1501] A means for receiving settings changed by the user based on the aforementioned automated proposal,

[1502] Means for controlling a smart environment device to execute the received settings,

[1503] A means for detecting customer emotional data (facial expressions, voice tone, behavior) within a physical store,

[1504] A means of analyzing the aforementioned emotional data and notifying store staff of specific product suggestions or environmental adjustments,

[1505] A system that includes this.

[1506] (Claim 2)

[1507] The system according to claim 1, wherein the aforementioned living environment data includes visual sensor information, audio sensor information, and behavioral records.

[1508] (Claim 3)

[1509] The system according to claim 1, wherein the analysis means detects the emotional state of users and customers using a machine learning algorithm and an emotion analysis engine. [Explanation of Symbols]

[1510] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving data about the user's living environment, A means for analyzing the aforementioned living environment data to generate optimal automation suggestions for the user, A means for notifying the user of the generated automation proposal, A means for receiving settings changed by the user based on the aforementioned automated proposal, Means for controlling a smart home device to execute the received settings, A system that includes this.

2. The system according to claim 1, wherein the aforementioned living environment data includes camera information, activity records, and floor plan information of the residence.

3. The system according to claim 1, wherein the analysis means learns the user's lifestyle patterns using a machine learning algorithm.

Images