System

Abstract

A system is provided.SOLUTION: A system, comprising: means for collecting sensor data of a user; means for analyzing the collected sensor data of the user to identify a behavior pattern of the user; means for training a machine learning model based on an analysis result to predict a future behavior of the user; and means for automatically adjusting a setting of a smart home device based on a prediction result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by this invention is to automate environmental adjustments in a smart home that are tailored to the user's personal preferences and lifestyle patterns, thereby improving user comfort and optimizing energy consumption in the home. Specifically, the object is to provide a system that can provide settings that correspond to each scenario—at home, out, and at bedtime—in real time, eliminating the need for the user to frequently change settings manually. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system including a means for collecting user sensor data, a means for analyzing the collected user sensor data to identify the user's behavioral patterns, a means for training a machine learning model based on the analysis results to predict the user's future behavior, and a means for automatically adjusting settings of smart home devices based on the prediction results. This system automatically applies different home environment settings for when the user is at home, away from home, and at bedtime, thereby improving user comfort and energy efficiency. Furthermore, by adjusting lighting, temperature, and music settings, it is possible to provide a customized environment tailored to the user's individual needs.

[0006] "Sensor data" is data about user behavior and the environment collected by sensors in the home.

[0007] A "behavioral pattern" is a tendency of time or behavior that is identified by the user's lifestyle habits and repetitive behavior.

[0008] "Analysis" is the process of identifying user behavior patterns by analyzing collected sensor data.

[0009] A "machine learning model" is a collection of algorithms and data processing methods used to predict user behavior patterns.

[0010] "Prediction" refers to using machine learning models to predict future user behavior.

[0011] "Smart home devices" is a general term for household electrical and electronic devices that can be controlled via the Internet or an internal network.

[0012] "Automatically adjust settings" refers to automatically changing the settings of smart home devices, such as lighting, temperature, and music, based on prediction results.

[0013] "Comfort" is the state of the home environment that provides physical and psychological satisfaction to the user.

[0014] "Energy optimization" means minimizing the amount of energy consumed while maintaining user comfort. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that collects and analyzes user sensor data to identify behavioral patterns, uses this data to predict future behavior using a machine learning model, and automatically adjusts the settings of smart home devices based on the prediction results.

[0037] The system consists of the following main components:

[0038] 1. User data collection module

[0039] 2. Data Analysis Module

[0040] 3. Machine Learning Models

[0041] 4. Smart Home Device Control Module

[0042] 1. User data collection module

[0043] The device collects real-time sensor data such as temperature, illuminance, and sound from sensors installed in the room and from the user's smartphone, and then transmits this data to a server via the network.

[0044] 2. Data Analysis Module

[0045] The server passes the collected sensor data to a data analysis module, which uses this data to identify user behavior patterns. For example, if a user is in the living room between 7 and 9 in the morning, this behavior pattern will be identified.

[0046] 3. Machine Learning Models

[0047] The server trains a machine learning model based on the behavioral patterns obtained from the data analysis module. This model can learn from past data and predict future behavior. For example, it can learn that a user wakes up at 7 a.m. every morning and eats breakfast in the living room.

[0048] 4. Smart Home Device Control Module

[0049] The server then operates the smart home device control module based on the prediction results. This module automatically adjusts environmental settings such as lighting, temperature, and music according to whether the user is at home, away from home, or asleep. For example, if the server predicts that the user is at home, it can turn on the lights, set the room temperature to 22 degrees, and play relaxing music.

[0050] Specific examples

[0051] For example, if a device's sensor detects that the user is in the living room at 7:00 AM, the following process occurs: The server receives this data, and the data analysis module identifies the user's behavioral patterns. As a result, it is determined that the user is active in the living room between 7:00 and 9:00 AM. This data is used to train a machine learning model, which predicts that the user will be active in the living room at the same time the next day. Based on this prediction, the server uses the smart home device control module to turn on the lights before 7:00 AM, set the temperature to 22 degrees, and play relaxing music. This allows the user to wake up in a comfortable environment.

[0052] Similarly, when the device detects that the user has left the house, the server will turn off the lights and set the temperature to 18 degrees to save energy. When it's time to go to bed, the server will turn off the lights, set the temperature to 20 degrees, and play sleep-inducing music. In this way, the system can automatically provide the optimal environment tailored to the user's lifestyle.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The device collects user behavior data in real time from various sensors in the home (temperature sensors, motion sensors, light sensors, etc.).

[0056] Step 2:

[0057] The terminals transmit the collected sensor data to a server via a network, where the data is accumulated in real time.

[0058] Step 3:

[0059] The server stores the received sensor data in the user's data collection module, which stores and manages the data.

[0060] Step 4:

[0061] The server analyzes the accumulated data using a data analysis module, which identifies user behavior patterns, such as being in the living room at a specific time.

[0062] Step 5:

[0063] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0064] Step 6:

[0065] As users go about their daily lives, new sensor data is generated, and this data is collected again by the device.

[0066] Step 7:

[0067] The server inputs new data into a machine learning model to predict the user's future behavior, such as whether the user will be at home or out at a certain time the next day.

[0068] Step 8:

[0069] The server then operates a smart home device control module based on the prediction results, which automatically adjusts settings such as lighting, heating, and music.

[0070] Step 9:

[0071] The device receives the new settings from the server and applies them to the smart home devices, thus preparing an optimal environment for the predicted behavior.

[0072] Step 10:

[0073] Users enjoy an optimized home environment: for example, lights are turned on, the temperature is set to 22 degrees, and relaxing music is played when they are at home, and when they leave, the lights are turned off to optimize energy consumption.

[0074] By repeating these steps, the system continuously learns and adapts to the user's lifestyle patterns, achieving both a comfortable and energy-efficient home environment.

[0075] Example 1

[0076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0077] Conventional smart home systems cannot accurately grasp user behavior patterns and can only set fixed settings for general time periods and situations, which means they do not fully achieve user comfort or energy efficiency.In addition, there is a lack of systems that can automatically collect and analyze sensor data, identify behavior patterns, and predict future behavior as a series of processes, making it difficult to respond to specific user needs.

[0078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0079] In this invention, the server includes means for transmitting user sensor data over a network, means for analyzing the received sensor data to identify user behavior patterns, means for training a machine learning model based on the identified behavior patterns to predict the user's future behavior, and means for automatically adjusting the settings of smart home devices based on the prediction results, thereby enabling detailed environmental settings based on user behavior and improving comfort and energy efficiency.

[0080] "Sensor data" refers to information such as temperature, light level, sound, and movement acquired by sensors to detect user behavior and environmental conditions.

[0081] A "network" is a communications infrastructure used by multiple terminals to exchange data with each other.

[0082] "Data analytics" is a technology for processing collected sensor data and identifying user behavior patterns and environmental conditions.

[0083] "Behavioral patterns" refer to the tendency of a user to take certain actions or engage in certain activities at certain times of the day and in certain places.

[0084] A "machine learning model" is an algorithm or program that learns from past data and predicts future behavior and states.

[0085] "Future behavior prediction" is the process of estimating what actions a user will take in the future based on data collected and analyzed in the past.

[0086] "Smart home devices" are electrical and electronic devices for the home that can be remotely or automatically controlled via a network.

[0087] "Automatically adjusting settings" means that the system automatically changes the operating state of smart home devices based on predictions.

[0088] The system of the present invention collects user sensor data, analyzes it to identify behavioral patterns, uses machine learning models to predict future behavior, and automatically adjusts the settings of smart home devices based on those predictions. The main components of the system are:

[0089] 1. User data collection module

[0090] 2. Data Analysis Module

[0091] 3. Machine Learning Models

[0092] 4. Smart Home Device Control Module

[0093] 1. User data collection module

[0094] The device collects real-time sensor data such as temperature, light, sound, and movement from various sensors installed in the room and from the user's smartphone. For example, it can detect and collect data on the temperature, light, and decibel levels of sound in a living room. The device then transmits the collected data to a server via a network. Specific hardware used includes a temperature sensor, light sensor, and microphone.

[0095] 2. Data Analysis Module

[0096] The server receives sensor data sent from the device and passes it to the data analysis module. The data analysis module preprocesses the received data, corrects outliers and missing values, and identifies user behavior patterns. For example, it can identify a pattern in which a user is active in the living room between 7 and 9 every morning. Data analysis tools such as Python and R can be used for the analysis.

[0097] 3. Machine Learning Models

[0098] The server trains a machine learning model based on the behavioral patterns identified by the data analysis module. The machine learning model can learn from past behavioral pattern data and predict future behavior. For example, it can predict whether the user will be in the living room at 7:00 the next day. Machine learning frameworks such as TensorFlow and Scikit-learn can be used for machine learning.

[0099] 4. Smart Home Device Control Module

[0100] The server runs a smart home device control module based on behavioral predictions made by the machine learning model. This module automatically adjusts settings such as lighting, temperature, and music according to the user's predicted behavior. For example, if the module predicts that the user will be in the living room at 7 a.m., it can turn on the lights, set the temperature to 22 degrees, and play relaxing music. Smart home devices include smart lights, smart thermostats, and smart speakers.

[0101] Specific examples

[0102] For example, if a device's sensor detects that the user is in the living room at 7:00 AM, the following process occurs: The server receives this data, and the data analysis module identifies the user's behavioral patterns. As a result, it is determined that the user is active in the living room between 7:00 and 9:00 AM. This data is used to train a machine learning model, which predicts that the user will be active in the living room at the same time the next day. Based on this prediction, the server uses the smart home device control module to turn on the lights before 7:00 AM, set the temperature to 22 degrees, and play relaxing music. This allows the user to wake up in a comfortable environment.

[0103] Examples of prompt statements

[0104] "Please explain how a system can learn a user's behavioral pattern, such as waking up at 7am every morning and eating breakfast in the living room, and automatically adjust the settings of its smart home devices if it predicts the same behavior the next day."

[0105] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0106] Step 1:

[0107] The device collects sensor data such as temperature, illuminance, sound, and movement in real time from sensors installed in the room and from the user's smartphone. Specifically, it detects that the temperature in the living room is 22 degrees, the illuminance is 300 lux, and the sound decibel level is 40 dB. This sensor data is the input. The device acquires this data at regular intervals (for example, every second) and compiles it into a data set.

[0108] Step 2:

[0109] The device sends the collected sensor data to the server via the network. The sent data set includes all acquired sensor data. This transmission is the output. Specifically, the data is sent from the device to the server using an HTTP POST request.

[0110] Step 3:

[0111] The server receives sensor data sent from the device and passes it to the data analysis module. The received data is the input, and the processing of the data passed to the data analysis module is the output. Specifically, the server stores the sensor data in a database, which is then read by the data analysis module.

[0112] Step 4:

[0113] The server's data analysis module preprocesses the received sensor data. The input for preprocessing is raw data, and the output is clean data. Specifically, it corrects outliers and fills in missing values. For example, if the temperature shows an abnormally high value, it will exclude that data point.

[0114] Step 5:

[0115] The server's data analysis module analyzes the preprocessed data and identifies user behavior patterns. The input is clean data, and the output is the identified behavior patterns. Specifically, it identifies that a user is performing a specific activity at a specific time (e.g., being in the living room at 7 a.m.) based on time-series data.

[0116] Step 6:

[0117] The server trains a machine learning model based on the identified behavioral patterns. The input is the behavioral pattern data, and the output is the trained model. Specifically, the server uses past behavioral pattern data to train the model using a machine learning algorithm (e.g., TensorFlow or Scikit-learn).

[0118] Step 7:

[0119] The server uses the trained machine learning model to predict future behavior. The input is the trained model and new sensor data, and the output is the predicted result. Specifically, it predicts the behavior pattern for the next day (e.g., the user will be in the living room from 7:00 to 9:00).

[0120] Step 8:

[0121] The server issues instructions to the smart home device control module based on future behavior predictions. The input is the prediction result, and the output is the device control command. Specifically, it instructs the smart home device to turn on the lights before 7:00, set the temperature to 22 degrees, and play relaxing music.

[0122] This allows users to live their daily lives in a comfortable environment.

[0123] (Application example 1)

[0124] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] In conventional virtual stores, it was difficult to efficiently collect and analyze user behavior patterns and optimize product suggestions and layouts in real time based on that data. Furthermore, the lack of automatic adjustments to store design based on predicted user behavior limited improvements to the quality of the user experience. For this reason, there was a need to develop a system that could utilize user behavior data to optimize product suggestions and layouts within a virtual store.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0127] In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying user behavior patterns, means for training a machine learning model based on the analysis results and predicting the user's future behavior, and means for adjusting the layout and product proposals of the virtual store based on the prediction results, thereby making it possible to optimize the product proposals and layout of the virtual store in real time based on the user behavior data.

[0128] "User sensor data" refers to information about a user's behavior and environment collected from smart devices and sensor devices.

[0129] A "behavioral pattern" is a series of specific actions that indicate a user's habits or tendencies, derived from analyzing collected user data.

[0130] A "machine learning model" is an algorithm or system that learns from data and predicts users' future behavior and tendencies.

[0131] "Predictive tuning" is the process of automatically changing system or environmental settings based on user behavior predicted by a machine learning model.

[0132] A "virtual store" is a commercial establishment operated on the Internet or in a virtual environment, where users can purchase products or use services online.

[0133] "Product suggestion" is the act of presenting the most suitable product to a user based on user behavioral data and predictive data.

[0134] "Real time" refers to ongoing or near-simultaneous events and implies immediate response without delay.

[0135] "Layout optimization" is the process of changing the product placement and design within a store to make it the most efficient and user-friendly, based on user behavior patterns and predictive data.

[0136] This invention is a system that collects user sensor data, analyzes it to identify user behavior patterns, predicts future behavior using machine learning models, and adjusts the layout and product recommendations of a virtual store based on the predictions. The system consists of the following main components:

[0137] 1. User data collection module:

[0138] Sensor data (such as the time the user looks and the product information viewed) is collected in real time from the smart device (e.g., smart glasses) used by the user. This data is sent to a server via a network.

[0139] 2. Data Analysis Module:

[0140] The server passes the collected sensor data to a data analysis module, which uses this data to identify user behavior patterns, such as when a user gazes at a particular product for a long period of time.

[0141] 3. Machine learning models:

[0142] The server trains a machine learning model (using TensorFlow as an example) based on the behavioral patterns obtained from the data analysis module. This model can learn from past data and predict future behavior. For example, it can learn patterns of users frequently viewing certain products and predict which products they are likely to view next.

[0143] 4. Virtual store layout and product proposal adjustment module:

[0144] The server then adjusts the virtual store's layout and product recommendations based on the prediction results. This module changes product placement and design so that users can experience the optimal products and layout in the virtual store. For example, it prioritizes displaying products that the user is looking at and places related products nearby.

[0145] The specific hardware and software used

[0146] Hardware: Smart glasses

[0147] Software: Machine learning model (TensorFlow), data analysis module (Python)

[0148] Specific examples

[0149] For example, imagine a user wearing smart glasses visits a virtual store. The smart glasses' sensors collect the user's gaze data and send it to a server. The server analyzes this data and recognizes that the user is looking at a particular product for a long time. Based on this behavioral pattern, a machine learning model predicts the user's next move. Based on this prediction, the server adjusts the layout of the virtual store to prioritize displaying related products in the user's field of view.

[0150] Prompt Sentence Examples

[0151] "Predict what product a user is likely to look at next in a section where they look for more than 10 seconds (e.g., the TV section). Also, recommend store layout changes based on that prediction."

[0152] In this way, the present invention allows for real-time optimization of the virtual store environment based on user behavior.

[0153] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0154] Step 1:

[0155] The device collects sensor data from the user through the smart glasses, specifically, information on the time the user looks and the products they look at in real time, and sends this data to a server via a network.

[0156] Input: User gaze data, viewed product information

[0157] Output: Sending sensor data to the server

[0158] Step 2:

[0159] The server passes the received sensor data to a data analysis module, which analyzes the sensor data and identifies user behavior patterns, such as when a user gazes at a particular product for a long period of time.

[0160] Input: Sensor data sent from the device

[0161] Output: User behavior patterns

[0162] Step 3:

[0163] The server trains a machine learning model based on the behavioral patterns obtained from the data analysis module. This model learns from past data and has the ability to predict future behavior, such as which products a user is likely to look at next.

[0164] Input: User behavior patterns

[0165] Output: Updates to the machine learning model and predictions of future behavior

[0166] Step 4:

[0167] The server adjusts the virtual store's layout and product suggestions based on the results of the machine learning model. Specifically, it changes product placement and suggestions based on predicted behavior. For example, it prioritizes displaying products that the user is looking at and places related products nearby.

[0168] Input: Prediction results of machine learning model

[0169] Output: Coordinated virtual store layout and product suggestions

[0170] Step 5:

[0171] The server inputs prompts into the generative AI model to enable advanced product suggestions and store layout adjustments. For example, the server inputs a prompt such as, "Predict the next product that the user is likely to look at in the section where they have looked for more than 10 seconds."

[0172] Input: prompt sentence in generative AI model

[0173] Output: Suggested results from the generative AI model

[0174] Through the above processing steps, the present invention makes it possible to optimize the virtual store environment in real time based on user behavior data.

[0175] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0176] The present invention is a system that uses user sensor data to recognize emotions and automatically adjusts the settings of smart home devices based on those emotions, thereby further improving user comfort.

[0177] The system consists of the following main components:

[0178] 1. User data collection module

[0179] 2. Data Analysis Module

[0180] 3. Machine Learning Models

[0181] 4. Emotion Engine

[0182] 5. Smart Home Device Control Module

[0183] 1. User data collection module

[0184] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors in the home (temperature, motion, light, voice recognition, etc.), and transmits this data to a server via the network.

[0185] 2. Data Analysis Module

[0186] The server passes the collected sensor data to a data analysis module, which uses the data to identify user behavior patterns, such as where the user is at a particular time and what actions they are taking.

[0187] 3. Machine Learning Models

[0188] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0189] 4. Emotion Engine

[0190] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0191] 5. Smart Home Device Control Module

[0192] The server operates a smart home device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module automatically adjusts environmental settings according to the user's emotional state. For example, if the user is feeling stressed, it changes the lights to softer colors and plays relaxing music.

[0193] Specific examples

[0194] For example, if the device detects the user's morning activities and uses sensors to collect their tone of voice and facial expressions, the server receives this data and uses the data analysis module to identify the user's behavioral patterns. At the same time, the emotion engine analyzes the data and recognizes that the user is in a bad mood that morning.

[0195] As a result, the server uses machine learning models to predict that the user has a specific behavioral pattern each morning and recognizes that the user's current emotional state is bad. Based on this information, the smart home device control module activates, changing the lighting to softer, warmer colors and playing relaxing music.

[0196] Similarly, if the device collects sensor data during the user's after-work hours and the emotion engine detects that the user is feeling stressed, the server can respond by adjusting settings to dim the lights and play quiet music, allowing the user to enjoy a relaxing environment as soon as they get home.

[0197] As described above, the present invention, combined with an emotion engine, makes it possible to recognize the user's emotional state and automatically provide a home environment that responds to that state, thereby further improving comfort and optimizing energy efficiency within the home.

[0198] The processing flow will be explained below.

[0199] Step 1:

[0200] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors in the home (temperature sensors, motion sensors, light sensors, voice recognition sensors, etc.).

[0201] Step 2:

[0202] The terminals transmit the collected sensor data to a server via a network, where the data is accumulated in real time.

[0203] Step 3:

[0204] The server stores the received sensor data in the user's data collection module, which stores and manages the data.

[0205] Step 4:

[0206] The server analyzes the accumulated data using a data analysis module, which identifies user behavior patterns, such as being in the living room at a particular time.

[0207] Step 5:

[0208] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0209] Step 6:

[0210] As users go about their daily lives, new sensor data is generated, and this data is collected again by the device.

[0211] Step 7:

[0212] The server inputs new data into a machine learning model to predict the user's future behavior, such as whether the user will be at home or out at a certain time the next day.

[0213] Step 8:

[0214] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0215] Step 9:

[0216] The server operates a smart home device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model, and this module automatically adjusts environmental settings according to the user's emotional state.

[0217] Step 10:

[0218] The device receives new settings from the server and applies them to smart home devices, preparing an optimal environment for predicted behaviors and emotional states.

[0219] Step 11:

[0220] Users enjoy an optimized home environment: for example, if the user is feeling stressed, the lights will change to softer colors and relaxing music will play, and when the user leaves the house, the lights will be turned off to optimize energy consumption.

[0221] By repeating these steps, the system continuously learns and adapts to the user's lifestyle patterns and emotional state, a process that helps to balance comfort and energy efficiency in the home environment.

[0222] Example 2

[0223] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0224] While modern smart home technologies have the ability to automatically adjust a user's home environment, this is primarily based on time of day or preset scenarios. There is currently no system that can adjust the home environment in real time according to the user's emotions and behavior, and this has left the improvement of user comfort and optimization of energy efficiency unsatisfactory.

[0225] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying the user's behavioral pattern, means for training a machine learning model based on the analysis results and predicting the user's future behavior, means for analyzing emotion-related data such as the user's tone of voice and facial expression and recognizing the user's emotional state, and means for automatically adjusting the settings of the smart device based on the recognized user's emotional state and the predicted behavioral pattern. This makes it possible to optimally adjust the home environment in real time based on the user's emotions and behavior.

[0226] "User sensor data" is data related to a user, such as temperature, motion, light level, voice tone, and facial expressions.

[0227] "Analysis means" refers to the technology and devices used to identify user behavior patterns based on collected sensor data.

[0228] A "machine learning model" is a model that learns a user's daily behavioral patterns based on collected and analyzed data and predicts future behavior.

[0229] "Voice tone" is emotion-related data extracted from audio signals, such as voice pitch, intonation, and tempo.

[0230] "Facial expression data" is data that indicates information related to emotions extracted from the facial expression of the user.

[0231] "Emotional state" is the user's mental and emotional state as perceived from the analyzed data.

[0232] "Smart devices" are internet-connected devices and systems used to control and regulate the home environment.

[0233] "Automatic adjustment means" refers to a means for automatically changing or adjusting the settings of a smart device based on the user's emotional state and behavioral patterns.

[0234] The present invention is a system that uses user sensor data to recognize emotions and automatically adjusts smart device settings based on those emotions, further improving user comfort and energy efficiency in the home.

[0235] System Configuration

[0236] The system consists of the following main components:

[0237] 1. User data collection module

[0238] 2. Data Analysis Module

[0239] 3. Machine Learning Models

[0240] 4. Emotion Engine

[0241] 5. Smart Device Control Module

[0242] 1. User data collection module

[0243] The device collects data related to the user's behavior and emotions from various sensors in the home (temperature sensor, motion sensor, light sensor, voice recognition sensor, etc.). For example, a temperature sensor installed in the living room measures the current room temperature, and a motion sensor detects the user's movements. The collected data is sent to a server in real time via the network.

[0244] 2. Data Analysis Module

[0245] The server receives sensor data sent from the device and passes it to the data analysis module. Using the Python Pandas library, the received data is formatted and analyzed to identify user behavior patterns. For example, it analyzes information such as which room a user is in at a specific time and what actions they are taking.

[0246] 3. Machine Learning Models

[0247] The server trains a machine learning model based on the results of the data analysis. This model learns the user's daily behavioral patterns and is able to predict future behavior. Machine learning libraries such as Scikit-learn and TensorFlow are used for training. This allows the server to predict future behavior with high accuracy based on the user's past data.

[0248] 4. Emotion Engine

[0249] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine. The emotion engine then analyzes this data using Google Cloud's emotion analysis API and Amazon Rekognition to recognize the user's emotional state in real time. For example, it uses voice recognition to read stress levels from the user's voice and image processing technology to recognize emotions from facial expressions.

[0250] 5. Smart Device Control Module

[0251] The server operates the smart device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module then configures the environment optimally to reflect the user's emotions. For example, if the server determines that the user is feeling stressed, it can instruct the smart lighting system to change the lighting to a soft, warm color and play relaxing music.

[0252] Specific examples

[0253] Morning Cases

[0254] The device uses sensors to collect data on the user's tone of voice and facial expressions, which are then sent to the server. The server processes this data with a data analysis module and determines that the user is in a bad mood. At the same time, a machine learning model predicts that the user has a specific behavioral pattern each morning. Based on this information, the server can activate the smart device control module to change the lighting to a warmer hue and play relaxing music to soothe the user's bad mood.

[0255] Prompt Sentence Examples

[0256] "This system uses user sensor data to recognize emotions and automatically adjust the settings on a smart device. Please explain the process flow of this system in concrete steps. Also, please explain the specific operations from collecting sensor data to adjusting the environment."

[0257] As described above, this system can recognize the user's emotional state in real time and automatically optimize the home environment based on that information, thereby improving user comfort and optimizing energy efficiency within the home.

[0258] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0259] Step 1: Data collection

[0260] The device uses various sensors in the home (such as temperature, motion, illuminance, and voice recognition sensors) to collect data related to the user's behavior and emotions. Specifically, the temperature sensor in the living room measures the current room temperature, and the motion sensor detects the user's movements. The voice recognition sensor captures the user's tone of voice, and the camera collects facial expression data. This data is sent from the device to the server in real time.

[0261] Input: Sensor data (temperature, movement, light, voice tone, facial expression)

[0262] Output: Raw data sent to the server

[0263] Step 2: Data reception and processing

[0264] The server receives raw data sent from the terminal. The received data is passed to the data analysis module, where preprocessing is performed. Specifically, the Python Pandas library is used to perform tasks such as filling in missing values, removing outliers, and integrating data to ensure consistency across time axes.

[0265] Input: Raw data sent from the terminal

[0266] Output: Preprocessed clean data

[0267] Step 3: Behavioral pattern analysis

[0268] The server analyzes user behavior patterns based on the clean data. The data analysis module identifies which room a user is in at what time of day and what actions they take. For example, it recognizes a pattern that a user is often in the kitchen between 8 and 9 a.m. This analysis is performed using Python's Pandas and NumPy.

[0269] Input: Preprocessed clean data

[0270] Output: User behavior patterns

[0271] Step 4: Train the machine learning model

[0272] The server trains a machine learning model based on the results of the behavioral pattern analysis. Using libraries such as Scikit-learn and TensorFlow, the model is fed with behavioral pattern data to predict future behavior, such as the probability that a user will be in a particular room at a particular time.

[0273] Input: User behavior patterns

[0274] Output: A trained machine learning model

[0275] Step 5: Sentiment Analysis

[0276] The server inputs the user's tone of voice and facial expression data into the emotion engine. This emotion engine uses Google Cloud's emotion analysis API and Amazon Rekognition to analyze the data and recognize the user's emotional state. For example, it can determine stress levels from voice data and identify joy, anger, sadness, and happiness from facial expression data.

[0277] Input: User's voice tone, facial expression data

[0278] Output: User's emotional state

[0279] Step 6: Automatic settings adjustment

[0280] The server operates a smart device control module based on the results of the emotion engine (user's emotional state) and the predictions of the machine learning model (future behavioral patterns). This module automatically adjusts environmental settings based on the user's emotional state and predicted behavior. For example, if the user is feeling stressed, the server instructs the smart lighting system to change the lighting to warmer colors and play relaxing music.

[0281] Input: User's emotional state, prediction results of machine learning model

[0282] Output: Adjusted smart device settings

[0283] (Application example 2)

[0284] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0285] This system solves the problem of improving user comfort by recognizing the user's emotions and behavior in real time while in the car and automatically adjusting the in-car environment based on that. Conventional in-car environment adjustment systems do not take the user's emotions and behavior into account, so they can only provide uniform settings and are unable to provide an optimal environment for each individual user. This means that users have to manually adjust the environment settings, which is inconvenient.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0287] In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying the user's behavioral patterns, means for training a machine learning model based on the analysis results and predicting the user's future behavior, and means for automatically adjusting the in-vehicle environmental settings based on the prediction results, thereby making it possible to optimally adjust the in-vehicle environment in real time according to the user's emotions and behavior while riding.

[0288] "Sensor data" refers to data collected from various sensors used to detect user emotions and behavior.

[0289] "Behavioral patterns" indicate the tendencies and habits of a user's behavior over a certain period of time, and are analyzed from sensor data.

[0290] A "machine learning model" is an algorithm that is trained using collected data to predict a user's future behavior.

[0291] "In-vehicle environmental settings" refers to the settings of the vehicle's interior environment (lighting, air conditioning, music, etc.) that are adjusted to improve the user's comfort.

[0292] "At the time of boarding" refers to the time when the user gets into the vehicle.

[0293] "When exiting the vehicle" refers to the point at which the user exits the vehicle.

[0294] "In motion" refers to the time the vehicle is in motion, whether driven or autonomous.

[0295] "Lighting" refers to the brightness and color settings of lights and lighting equipment installed inside the vehicle.

[0296] "Air conditioning" refers to the settings of the equipment used to regulate the temperature and humidity inside the vehicle.

[0297] "Music" refers to songs and sounds played in the vehicle, selected according to the user's emotional state.

[0298] This invention is a system that recognizes the user's emotions and behavior in real time while riding and automatically adjusts the in-car environment based on that. The system includes the following main components:

[0299] 1. User data collection module

[0300] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors installed in the vehicle (temperature sensors, motion sensors, illuminance sensors, voice recognition sensors, etc.), and transmits this data to a server via the network.

[0301] 2. Data Analysis Module

[0302] The server passes the collected sensor data to a data analysis module, which uses the data to identify user behavior patterns, such as where the user is at a particular time and what actions they are taking.

[0303] 3. Machine Learning Models

[0304] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0305] 4. Emotion Engine

[0306] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0307] 5. Vehicle climate control module

[0308] The server operates the vehicle's environmental control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module automatically adjusts the vehicle's environmental settings (lighting, air conditioning, music, etc.) according to the user's emotional state. For example, if the user is feeling stressed, the module will change the lighting to softer colors and play relaxing music.

[0309] Specific examples

[0310] For example, if the device senses the user's emotions while riding and collects their tone of voice and facial expressions using sensors, the server receives this data and uses the data analysis module to identify the user's behavioral patterns. At the same time, the emotion engine analyzes the data and recognizes that the user is feeling stressed at that time.

[0311] As a result, the server uses machine learning models to predict specific behavioral patterns during the ride and recognizes that the user's current emotional state is stressed. Based on this information, the vehicle's climate control module activates, changing the lighting to softer, warmer colors and playing relaxing music.

[0312] Similarly, if the device collects sensor data while the user is traveling and the emotion engine recognizes that the user is feeling happy, the server can respond by adjusting the settings to brighten the lights and play upbeat music, helping the user feel more comfortable while traveling.

[0313] Prompt Sentence Examples

[0314] "If a user is feeling stressed, how can we adjust the in-car environment to make them more relaxed?"

[0315] "Please explain how to optimize the in-car environment for a happy user."

[0316] In this way, the system can be applied to improve user comfort in vehicles as well.

[0317] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0318] Step 1:

[0319] The terminal collects user behavioral data and emotional data (tone of voice, facial expressions, etc.) in real time from various sensors installed inside the vehicle (temperature sensors, motion sensors, illuminance sensors, voice recognition sensors, etc.). This sensor data becomes the input for the terminal. The input data undergoes initial processing, including preprocessing such as noise removal and format standardization. The terminal then transmits the processed sensor data to a server via the network.

[0320] Step 2:

[0321] The server receives the sensor data sent from the device and analyzes the user's behavioral patterns using a data analysis module. In this analysis, the collected data is treated as variables (time, location, behavior, etc.). An analysis algorithm is used to identify the behavioral patterns, and the specific behavioral patterns are output as the result.

[0322] Step 3:

[0323] The server trains a machine learning model based on the analysis results. The input data is behavioral patterns and associated emotional data. Using this data, the machine learning model learns the user's daily behavioral patterns. Training continues over multiple epochs, and the final output is a model that can predict the user's future behavior.

[0324] Step 4:

[0325] The server runs an emotion engine that uses machine learning models to predict the user's emotional state in real time. Whenever sensor data is input, the emotion engine analyzes the data and outputs the user's current emotional state.

[0326] Step 5:

[0327] The server combines the output of the emotion engine with the predictions of the machine learning model and sends them to the vehicle climate control module. The inputs are the driver's emotional state and predicted behavioral patterns, and the optimal settings for the vehicle's climate are output based on these. These settings are then output as signals that automatically adjust elements such as lighting, air conditioning, and music.

[0328] Step 6:

[0329] The vehicle climate control module automatically adjusts the in-car environment (lighting, air conditioning, music, etc.) based on signals sent from the server. Specifically, it changes the color and brightness of the lights, sets the appropriate temperature, and plays music that matches the user's emotional state, thereby improving user comfort in real time.

[0330] In this way, each step works in coordination to realize a system that provides an optimal in-car environment that matches the emotional state of the user while riding.

[0331] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0332] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0333] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0334] [Second embodiment]

[0335] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0336] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0337] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0338] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0339] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0340] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0341] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0342] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0343] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0344] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0345] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0346] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0347] The present invention is a system that collects and analyzes user sensor data to identify behavioral patterns, uses this data to predict future behavior using a machine learning model, and automatically adjusts the settings of smart home devices based on the prediction results.

[0348] The system consists of the following main components:

[0349] 1. User data collection module

[0350] 2. Data Analysis Module

[0351] 3. Machine Learning Models

[0352] 4. Smart Home Device Control Module

[0353] 1. User data collection module

[0354] The device collects real-time sensor data such as temperature, illuminance, and sound from sensors installed in the room and from the user's smartphone, and then transmits this data to a server via the network.

[0355] 2. Data Analysis Module

[0356] The server passes the collected sensor data to a data analysis module, which uses this data to identify user behavior patterns. For example, if a user is in the living room between 7 and 9 in the morning, this behavior pattern will be identified.

[0357] 3. Machine Learning Models

[0358] The server trains a machine learning model based on the behavioral patterns obtained from the data analysis module. This model can learn from past data and predict future behavior. For example, it can learn that a user wakes up at 7 a.m. every morning and eats breakfast in the living room.

[0359] 4. Smart Home Device Control Module

[0360] The server then operates the smart home device control module based on the prediction results. This module automatically adjusts environmental settings such as lighting, temperature, and music according to whether the user is at home, away from home, or asleep. For example, if the server predicts that the user is at home, it can turn on the lights, set the room temperature to 22 degrees, and play relaxing music.

[0361] Specific examples

[0362] For example, if a device's sensor detects that the user is in the living room at 7:00 AM, the following process occurs: The server receives this data, and the data analysis module identifies the user's behavioral patterns. As a result, it is determined that the user is active in the living room between 7:00 and 9:00 AM. This data is used to train a machine learning model, which predicts that the user will be active in the living room at the same time the next day. Based on this prediction, the server uses the smart home device control module to turn on the lights before 7:00 AM, set the temperature to 22 degrees, and play relaxing music. This allows the user to wake up in a comfortable environment.

[0363] Similarly, when the device detects that the user has left the house, the server will turn off the lights and set the temperature to 18 degrees to save energy. When it's time to go to bed, the server will turn off the lights, set the temperature to 20 degrees, and play sleep-inducing music. In this way, the system can automatically provide the optimal environment tailored to the user's lifestyle.

[0364] The processing flow will be explained below.

[0365] Step 1:

[0366] The device collects user behavior data in real time from various sensors in the home (temperature sensors, motion sensors, light sensors, etc.).

[0367] Step 2:

[0368] The terminals transmit the collected sensor data to a server via a network, where the data is accumulated in real time.

[0369] Step 3:

[0370] The server stores the received sensor data in the user's data collection module, which stores and manages the data.

[0371] Step 4:

[0372] The server analyzes the accumulated data using a data analysis module, which identifies user behavior patterns, such as being in the living room at a specific time.

[0373] Step 5:

[0374] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0375] Step 6:

[0376] As users go about their daily lives, new sensor data is generated, and this data is collected again by the device.

[0377] Step 7:

[0378] The server inputs new data into a machine learning model to predict the user's future behavior, such as whether the user will be at home or out at a certain time the next day.

[0379] Step 8:

[0380] The server then operates a smart home device control module based on the prediction results, which automatically adjusts settings such as lighting, heating, and music.

[0381] Step 9:

[0382] The device receives the new settings from the server and applies them to the smart home devices, thus preparing an optimal environment for the predicted behavior.

[0383] Step 10:

[0384] Users enjoy an optimized home environment: for example, lights are turned on, the temperature is set to 22 degrees, and relaxing music is played when they are at home, and when they leave, the lights are turned off to optimize energy consumption.

[0385] By repeating these steps, the system continuously learns and adapts to the user's lifestyle patterns, achieving both a comfortable and energy-efficient home environment.

[0386] Example 1

[0387] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0388] Conventional smart home systems cannot accurately grasp user behavior patterns and can only set fixed settings for general time periods and situations, which means they do not fully achieve user comfort or energy efficiency.In addition, there is a lack of systems that can automatically collect and analyze sensor data, identify behavior patterns, and predict future behavior as a series of processes, making it difficult to respond to specific user needs.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0390] In this invention, the server includes means for transmitting user sensor data over a network, means for analyzing the received sensor data to identify user behavior patterns, means for training a machine learning model based on the identified behavior patterns to predict the user's future behavior, and means for automatically adjusting the settings of smart home devices based on the prediction results, thereby enabling detailed environmental settings based on user behavior and improving comfort and energy efficiency.

[0391] "Sensor data" refers to information such as temperature, light level, sound, and movement acquired by sensors to detect user behavior and environmental conditions.

[0392] A "network" is a communications infrastructure used by multiple terminals to exchange data with each other.

[0393] "Data analytics" is a technology for processing collected sensor data and identifying user behavior patterns and environmental conditions.

[0394] "Behavioral patterns" refer to the tendency of a user to take certain actions or engage in certain activities at certain times of the day and in certain places.

[0395] A "machine learning model" is an algorithm or program that learns from past data and predicts future behavior and states.

[0396] "Future behavior prediction" is the process of estimating what actions a user will take in the future based on data collected and analyzed in the past.

[0397] "Smart home devices" are electrical and electronic devices for the home that can be remotely or automatically controlled via a network.

[0398] "Automatically adjusting settings" means that the system automatically changes the operating state of smart home devices based on predictions.

[0399] The system of the present invention collects user sensor data, analyzes it to identify behavioral patterns, uses machine learning models to predict future behavior, and automatically adjusts the settings of smart home devices based on those predictions. The main components of the system are:

[0400] 1. User data collection module

[0401] 2. Data Analysis Module

[0402] 3. Machine Learning Models

[0403] 4. Smart Home Device Control Module

[0404] 1. User data collection module

[0405] The device collects real-time sensor data such as temperature, light, sound, and movement from various sensors installed in the room and from the user's smartphone. For example, it can detect and collect data on the temperature, light, and decibel levels of sound in a living room. The device then transmits the collected data to a server via a network. Specific hardware used includes a temperature sensor, light sensor, and microphone.

[0406] 2. Data Analysis Module

[0407] The server receives sensor data sent from the device and passes it to the data analysis module. The data analysis module preprocesses the received data, corrects outliers and missing values, and identifies user behavior patterns. For example, it can identify a pattern in which a user is active in the living room between 7 and 9 every morning. Data analysis tools such as Python and R can be used for the analysis.

[0408] 3. Machine Learning Models

[0409] The server trains a machine learning model based on the behavioral patterns identified by the data analysis module. The machine learning model can learn from past behavioral pattern data and predict future behavior. For example, it can predict whether the user will be in the living room at 7:00 the next day. Machine learning frameworks such as TensorFlow and Scikit-learn can be used for machine learning.

[0410] 4. Smart Home Device Control Module

[0411] The server runs a smart home device control module based on behavioral predictions made by the machine learning model. This module automatically adjusts settings such as lighting, temperature, and music according to the user's predicted behavior. For example, if the module predicts that the user will be in the living room at 7 a.m., it can turn on the lights, set the temperature to 22 degrees, and play relaxing music. Smart home devices include smart lights, smart thermostats, and smart speakers.

[0412] Specific examples

[0413] For example, if a device's sensor detects that the user is in the living room at 7:00 AM, the following process occurs: The server receives this data, and the data analysis module identifies the user's behavioral patterns. As a result, it is determined that the user is active in the living room between 7:00 and 9:00 AM. This data is used to train a machine learning model, which predicts that the user will be active in the living room at the same time the next day. Based on this prediction, the server uses the smart home device control module to turn on the lights before 7:00 AM, set the temperature to 22 degrees, and play relaxing music. This allows the user to wake up in a comfortable environment.

[0414] Examples of prompt statements

[0415] "Please explain how a system can learn a user's behavioral pattern, such as waking up at 7am every morning and eating breakfast in the living room, and automatically adjust the settings of its smart home devices if it predicts the same behavior the next day."

[0416] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0417] Step 1:

[0418] The device collects sensor data such as temperature, illuminance, sound, and movement in real time from sensors installed in the room and from the user's smartphone. Specifically, it detects that the temperature in the living room is 22 degrees, the illuminance is 300 lux, and the sound decibel level is 40 dB. This sensor data is the input. The device acquires this data at regular intervals (for example, every second) and compiles it into a data set.

[0419] Step 2:

[0420] The device sends the collected sensor data to the server via the network. The sent data set includes all acquired sensor data. This transmission is the output. Specifically, the data is sent from the device to the server using an HTTP POST request.

[0421] Step 3:

[0422] The server receives sensor data sent from the device and passes it to the data analysis module. The received data is the input, and the processing of the data passed to the data analysis module is the output. Specifically, the server stores the sensor data in a database, which is then read by the data analysis module.

[0423] Step 4:

[0424] The server's data analysis module preprocesses the received sensor data. The input for preprocessing is raw data, and the output is clean data. Specifically, it corrects outliers and fills in missing values. For example, if the temperature shows an abnormally high value, it will exclude that data point.

[0425] Step 5:

[0426] The server's data analysis module analyzes the preprocessed data and identifies user behavior patterns. The input is clean data, and the output is the identified behavior patterns. Specifically, it identifies that a user is performing a specific activity at a specific time (e.g., being in the living room at 7 a.m.) based on time-series data.

[0427] Step 6:

[0428] The server trains a machine learning model based on the identified behavioral patterns. The input is the behavioral pattern data, and the output is the trained model. Specifically, the server uses past behavioral pattern data to train the model using a machine learning algorithm (e.g., TensorFlow or Scikit-learn).

[0429] Step 7:

[0430] The server uses the trained machine learning model to predict future behavior. The input is the trained model and new sensor data, and the output is the predicted result. Specifically, it predicts the behavior pattern for the next day (e.g., the user will be in the living room from 7:00 to 9:00).

[0431] Step 8:

[0432] The server issues instructions to the smart home device control module based on future behavior predictions. The input is the prediction result, and the output is the device control command. Specifically, it instructs the smart home device to turn on the lights before 7:00, set the temperature to 22 degrees, and play relaxing music.

[0433] This allows users to live their daily lives in a comfortable environment.

[0434] (Application example 1)

[0435] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0436] In conventional virtual stores, it was difficult to efficiently collect and analyze user behavior patterns and optimize product suggestions and layouts in real time based on that data. Furthermore, the lack of automatic adjustments to store design based on predicted user behavior limited improvements to the quality of the user experience. For this reason, there was a need to develop a system that could utilize user behavior data to optimize product suggestions and layouts within a virtual store.

[0437] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0438] In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying user behavior patterns, means for training a machine learning model based on the analysis results and predicting the user's future behavior, and means for adjusting the layout and product proposals of the virtual store based on the prediction results, thereby making it possible to optimize the product proposals and layout of the virtual store in real time based on the user behavior data.

[0439] "User sensor data" refers to information about a user's behavior and environment collected from smart devices and sensor devices.

[0440] A "behavioral pattern" is a series of specific actions that indicate a user's habits or tendencies, derived from analyzing collected user data.

[0441] A "machine learning model" is an algorithm or system that learns from data and predicts users' future behavior and tendencies.

[0442] "Predictive tuning" is the process of automatically changing system or environmental settings based on user behavior predicted by a machine learning model.

[0443] A "virtual store" is a commercial establishment operated on the Internet or in a virtual environment, where users can purchase products or use services online.

[0444] "Product suggestion" is the act of presenting the most suitable product to a user based on user behavioral data and predictive data.

[0445] "Real time" refers to ongoing or near-simultaneous events and implies immediate response without delay.

[0446] "Layout optimization" is the process of changing the product placement and design within a store to make it the most efficient and user-friendly, based on user behavior patterns and predictive data.

[0447] This invention is a system that collects user sensor data, analyzes it to identify user behavior patterns, predicts future behavior using machine learning models, and adjusts the layout and product recommendations of a virtual store based on the predictions. The system consists of the following main components:

[0448] 1. User data collection module:

[0449] Sensor data (such as the time the user looks and the product information viewed) is collected in real time from the smart device (e.g., smart glasses) used by the user. This data is sent to a server via a network.

[0450] 2. Data Analysis Module:

[0451] The server passes the collected sensor data to a data analysis module, which uses this data to identify user behavior patterns, such as when a user gazes at a particular product for a long period of time.

[0452] 3. Machine learning models:

[0453] The server trains a machine learning model (using TensorFlow as an example) based on the behavioral patterns obtained from the data analysis module. This model can learn from past data and predict future behavior. For example, it can learn patterns of users frequently viewing certain products and predict which products they are likely to view next.

[0454] 4. Virtual store layout and product proposal adjustment module:

[0455] The server then adjusts the virtual store's layout and product recommendations based on the prediction results. This module changes product placement and design so that users can experience the optimal products and layout in the virtual store. For example, it prioritizes displaying products that the user is looking at and places related products nearby.

[0456] The specific hardware and software used

[0457] Hardware: Smart glasses

[0458] Software: Machine learning model (TensorFlow), data analysis module (Python)

[0459] Specific examples

[0460] For example, imagine a user wearing smart glasses visits a virtual store. The smart glasses' sensors collect the user's gaze data and send it to a server. The server analyzes this data and recognizes that the user is looking at a particular product for a long time. Based on this behavioral pattern, a machine learning model predicts the user's next move. Based on this prediction, the server adjusts the layout of the virtual store to prioritize displaying related products in the user's field of view.

[0461] Prompt Sentence Examples

[0462] "Predict what product a user is likely to look at next in a section where they look for more than 10 seconds (e.g., the TV section). Also, recommend store layout changes based on that prediction."

[0463] In this way, the present invention allows for real-time optimization of the virtual store environment based on user behavior.

[0464] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0465] Step 1:

[0466] The device collects sensor data from the user through the smart glasses, specifically, information on the time the user looks and the products they look at in real time, and sends this data to a server via a network.

[0467] Input: User gaze data, viewed product information

[0468] Output: Sending sensor data to the server

[0469] Step 2:

[0470] The server passes the received sensor data to a data analysis module, which analyzes the sensor data and identifies user behavior patterns, such as when a user gazes at a particular product for a long period of time.

[0471] Input: Sensor data sent from the device

[0472] Output: User behavior patterns

[0473] Step 3:

[0474] The server trains a machine learning model based on the behavioral patterns obtained from the data analysis module. This model learns from past data and has the ability to predict future behavior, such as which products a user is likely to look at next.

[0475] Input: User behavior patterns

[0476] Output: Updates to the machine learning model and predictions of future behavior

[0477] Step 4:

[0478] The server adjusts the virtual store's layout and product suggestions based on the results of the machine learning model. Specifically, it changes product placement and suggestions based on predicted behavior. For example, it prioritizes displaying products that the user is looking at and places related products nearby.

[0479] Input: Prediction results of machine learning model

[0480] Output: Coordinated virtual store layout and product suggestions

[0481] Step 5:

[0482] The server inputs prompts into the generative AI model to enable advanced product suggestions and store layout adjustments. For example, the server inputs a prompt such as, "Predict the next product that the user is likely to look at in the section where they have looked for more than 10 seconds."

[0483] Input: prompt sentence in generative AI model

[0484] Output: Suggested results from the generative AI model

[0485] Through the above processing steps, the present invention makes it possible to optimize the virtual store environment in real time based on user behavior data.

[0486] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0487] The present invention is a system that uses user sensor data to recognize emotions and automatically adjusts the settings of smart home devices based on those emotions, thereby further improving user comfort.

[0488] The system consists of the following main components:

[0489] 1. User data collection module

[0490] 2. Data Analysis Module

[0491] 3. Machine Learning Models

[0492] 4. Emotion Engine

[0493] 5. Smart Home Device Control Module

[0494] 1. User data collection module

[0495] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors in the home (temperature, motion, light, voice recognition, etc.), and transmits this data to a server via the network.

[0496] 2. Data Analysis Module

[0497] The server passes the collected sensor data to a data analysis module, which uses the data to identify user behavior patterns, such as where the user is at a particular time and what actions they are taking.

[0498] 3. Machine Learning Models

[0499] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0500] 4. Emotion Engine

[0501] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0502] 5. Smart Home Device Control Module

[0503] The server operates a smart home device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module automatically adjusts environmental settings according to the user's emotional state. For example, if the user is feeling stressed, it changes the lights to softer colors and plays relaxing music.

[0504] Specific examples

[0505] For example, if the device detects the user's morning activities and uses sensors to collect their tone of voice and facial expressions, the server receives this data and uses the data analysis module to identify the user's behavioral patterns. At the same time, the emotion engine analyzes the data and recognizes that the user is in a bad mood that morning.

[0506] As a result, the server uses machine learning models to predict that the user has a specific behavioral pattern each morning and recognizes that the user's current emotional state is bad. Based on this information, the smart home device control module activates, changing the lighting to softer, warmer colors and playing relaxing music.

[0507] Similarly, if the device collects sensor data during the user's after-work hours and the emotion engine detects that the user is feeling stressed, the server can respond by adjusting settings to dim the lights and play quiet music, allowing the user to enjoy a relaxing environment as soon as they get home.

[0508] As described above, the present invention, combined with an emotion engine, makes it possible to recognize the user's emotional state and automatically provide a home environment that responds to that state, thereby further improving comfort and optimizing energy efficiency within the home.

[0509] The processing flow will be explained below.

[0510] Step 1:

[0511] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors in the home (temperature sensors, motion sensors, light sensors, voice recognition sensors, etc.).

[0512] Step 2:

[0513] The terminals transmit the collected sensor data to a server via a network, where the data is accumulated in real time.

[0514] Step 3:

[0515] The server stores the received sensor data in the user's data collection module, which stores and manages the data.

[0516] Step 4:

[0517] The server analyzes the accumulated data using a data analysis module, which identifies user behavior patterns, such as being in the living room at a particular time.

[0518] Step 5:

[0519] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0520] Step 6:

[0521] As users go about their daily lives, new sensor data is generated, and this data is collected again by the device.

[0522] Step 7:

[0523] The server inputs new data into a machine learning model to predict the user's future behavior, such as whether the user will be at home or out at a certain time the next day.

[0524] Step 8:

[0525] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0526] Step 9:

[0527] The server operates a smart home device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model, and this module automatically adjusts environmental settings according to the user's emotional state.

[0528] Step 10:

[0529] The device receives new settings from the server and applies them to smart home devices, preparing an optimal environment for predicted behaviors and emotional states.

[0530] Step 11:

[0531] Users enjoy an optimized home environment: for example, if the user is feeling stressed, the lights will change to softer colors and relaxing music will play, and when the user leaves the house, the lights will be turned off to optimize energy consumption.

[0532] By repeating these steps, the system continuously learns and adapts to the user's lifestyle patterns and emotional state, a process that helps to balance comfort and energy efficiency in the home environment.

[0533] Example 2

[0534] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0535] While modern smart home technologies have the ability to automatically adjust a user's home environment, this is primarily based on time of day or preset scenarios. There is currently no system that can adjust the home environment in real time according to the user's emotions and behavior, and this has left the improvement of user comfort and optimization of energy efficiency unsatisfactory.

[0536] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying the user's behavioral pattern, means for training a machine learning model based on the analysis results and predicting the user's future behavior, means for analyzing emotion-related data such as the user's tone of voice and facial expression and recognizing the user's emotional state, and means for automatically adjusting the settings of the smart device based on the recognized user's emotional state and the predicted behavioral pattern. This makes it possible to optimally adjust the home environment in real time based on the user's emotions and behavior.

[0537] "User sensor data" is data related to a user, such as temperature, motion, light level, voice tone, and facial expressions.

[0538] "Analysis means" refers to the technology and devices used to identify user behavior patterns based on collected sensor data.

[0539] A "machine learning model" is a model that learns a user's daily behavioral patterns based on collected and analyzed data and predicts future behavior.

[0540] "Voice tone" is emotion-related data extracted from audio signals, such as voice pitch, intonation, and tempo.

[0541] "Facial expression data" is data that indicates information related to emotions extracted from the facial expression of the user.

[0542] "Emotional state" is the user's mental and emotional state as perceived from the analyzed data.

[0543] "Smart devices" are internet-connected devices and systems used to control and regulate the home environment.

[0544] "Automatic adjustment means" refers to a means for automatically changing or adjusting the settings of a smart device based on the user's emotional state and behavioral patterns.

[0545] The present invention is a system that uses user sensor data to recognize emotions and automatically adjusts smart device settings based on those emotions, further improving user comfort and energy efficiency in the home.

[0546] System Configuration

[0547] The system consists of the following main components:

[0548] 1. User data collection module

[0549] 2. Data Analysis Module

[0550] 3. Machine Learning Models

[0551] 4. Emotion Engine

[0552] 5. Smart Device Control Module

[0553] 1. User data collection module

[0554] The device collects data related to the user's behavior and emotions from various sensors in the home (temperature sensor, motion sensor, light sensor, voice recognition sensor, etc.). For example, a temperature sensor installed in the living room measures the current room temperature, and a motion sensor detects the user's movements. The collected data is sent to a server in real time via the network.

[0555] 2. Data Analysis Module

[0556] The server receives sensor data sent from the device and passes it to the data analysis module. Using the Python Pandas library, the received data is formatted and analyzed to identify user behavior patterns. For example, it analyzes information such as which room a user is in at a specific time and what actions they are taking.

[0557] 3. Machine Learning Models

[0558] The server trains a machine learning model based on the results of the data analysis. This model learns the user's daily behavioral patterns and is able to predict future behavior. Machine learning libraries such as Scikit-learn and TensorFlow are used for training. This allows the server to predict future behavior with high accuracy based on the user's past data.

[0559] 4. Emotion Engine

[0560] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine. The emotion engine then analyzes this data using Google Cloud's emotion analysis API and Amazon Rekognition to recognize the user's emotional state in real time. For example, it uses voice recognition to read stress levels from the user's voice and image processing technology to recognize emotions from facial expressions.

[0561] 5. Smart Device Control Module

[0562] The server operates the smart device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module then configures the environment optimally to reflect the user's emotions. For example, if the server determines that the user is feeling stressed, it can instruct the smart lighting system to change the lighting to a soft, warm color and play relaxing music.

[0563] Specific examples

[0564] Morning Cases

[0565] The device uses sensors to collect data on the user's tone of voice and facial expressions, which are then sent to the server. The server processes this data with a data analysis module and determines that the user is in a bad mood. At the same time, a machine learning model predicts that the user has a specific behavioral pattern each morning. Based on this information, the server can activate the smart device control module to change the lighting to a warmer hue and play relaxing music to soothe the user's bad mood.

[0566] Prompt Sentence Examples

[0567] "This system uses user sensor data to recognize emotions and automatically adjust the settings on a smart device. Please explain the process flow of this system in concrete steps. Also, please explain the specific operations from collecting sensor data to adjusting the environment."

[0568] As described above, this system can recognize the user's emotional state in real time and automatically optimize the home environment based on that information, thereby improving user comfort and optimizing energy efficiency within the home.

[0569] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0570] Step 1: Data collection

[0571] The device uses various sensors in the home (such as temperature, motion, illuminance, and voice recognition sensors) to collect data related to the user's behavior and emotions. Specifically, the temperature sensor in the living room measures the current room temperature, and the motion sensor detects the user's movements. The voice recognition sensor captures the user's tone of voice, and the camera collects facial expression data. This data is sent from the device to the server in real time.

[0572] Input: Sensor data (temperature, movement, light, voice tone, facial expression)

[0573] Output: Raw data sent to the server

[0574] Step 2: Data reception and processing

[0575] The server receives raw data sent from the terminal. The received data is passed to the data analysis module, where preprocessing is performed. Specifically, the Python Pandas library is used to perform tasks such as filling in missing values, removing outliers, and integrating data to ensure consistency across time axes.

[0576] Input: Raw data sent from the terminal

[0577] Output: Preprocessed clean data

[0578] Step 3: Behavioral pattern analysis

[0579] The server analyzes user behavior patterns based on the clean data. The data analysis module identifies which room a user is in at what time of day and what actions they take. For example, it recognizes a pattern that a user is often in the kitchen between 8 and 9 a.m. This analysis is performed using Python's Pandas and NumPy.

[0580] Input: Preprocessed clean data

[0581] Output: User behavior patterns

[0582] Step 4: Train the machine learning model

[0583] The server trains a machine learning model based on the results of the behavioral pattern analysis. Using libraries such as Scikit-learn and TensorFlow, the model is fed with behavioral pattern data to predict future behavior, such as the probability that a user will be in a particular room at a particular time.

[0584] Input: User behavior patterns

[0585] Output: A trained machine learning model

[0586] Step 5: Sentiment Analysis

[0587] The server inputs the user's tone of voice and facial expression data into the emotion engine. This emotion engine uses Google Cloud's emotion analysis API and Amazon Rekognition to analyze the data and recognize the user's emotional state. For example, it can determine stress levels from voice data and identify joy, anger, sadness, and happiness from facial expression data.

[0588] Input: User's voice tone, facial expression data

[0589] Output: User's emotional state

[0590] Step 6: Automatic settings adjustment

[0591] The server operates a smart device control module based on the results of the emotion engine (user's emotional state) and the predictions of the machine learning model (future behavioral patterns). This module automatically adjusts environmental settings based on the user's emotional state and predicted behavior. For example, if the user is feeling stressed, the server instructs the smart lighting system to change the lighting to warmer colors and play relaxing music.

[0592] Input: User's emotional state, prediction results of machine learning model

[0593] Output: Adjusted smart device settings

[0594] (Application example 2)

[0595] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0596] This system solves the problem of improving user comfort by recognizing the user's emotions and behavior in real time while in the car and automatically adjusting the in-car environment based on that. Conventional in-car environment adjustment systems do not take the user's emotions and behavior into account, so they can only provide uniform settings and are unable to provide an optimal environment for each individual user. This means that users have to manually adjust the environment settings, which is inconvenient.

[0597] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0598] In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying the user's behavioral patterns, means for training a machine learning model based on the analysis results and predicting the user's future behavior, and means for automatically adjusting the in-vehicle environmental settings based on the prediction results, thereby making it possible to optimally adjust the in-vehicle environment in real time according to the user's emotions and behavior while riding.

[0599] "Sensor data" refers to data collected from various sensors used to detect user emotions and behavior.

[0600] "Behavioral patterns" indicate the tendencies and habits of a user's behavior over a certain period of time, and are analyzed from sensor data.

[0601] A "machine learning model" is an algorithm that is trained using collected data to predict a user's future behavior.

[0602] "In-vehicle environmental settings" refers to the settings of the vehicle's interior environment (lighting, air conditioning, music, etc.) that are adjusted to improve the user's comfort.

[0603] "At the time of boarding" refers to the time when the user gets into the vehicle.

[0604] "When exiting the vehicle" refers to the point at which the user exits the vehicle.

[0605] "In motion" refers to the time the vehicle is in motion, whether driven or autonomous.

[0606] "Lighting" refers to the brightness and color settings of lights and lighting equipment installed inside the vehicle.

[0607] "Air conditioning" refers to the settings of the equipment used to regulate the temperature and humidity inside the vehicle.

[0608] "Music" refers to songs and sounds played in the vehicle, selected according to the user's emotional state.

[0609] This invention is a system that recognizes the user's emotions and behavior in real time while riding and automatically adjusts the in-car environment based on that. The system includes the following main components:

[0610] 1. User data collection module

[0611] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors installed in the vehicle (temperature sensors, motion sensors, illuminance sensors, voice recognition sensors, etc.), and transmits this data to a server via the network.

[0612] 2. Data Analysis Module

[0613] The server passes the collected sensor data to a data analysis module, which uses the data to identify user behavior patterns, such as where the user is at a particular time and what actions they are taking.

[0614] 3. Machine Learning Models

[0615] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0616] 4. Emotion Engine

[0617] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0618] 5. Vehicle climate control module

[0619] The server operates the vehicle's environmental control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module automatically adjusts the vehicle's environmental settings (lighting, air conditioning, music, etc.) according to the user's emotional state. For example, if the user is feeling stressed, the module will change the lighting to softer colors and play relaxing music.

[0620] Specific examples

[0621] For example, if the device senses the user's emotions while riding and collects their tone of voice and facial expressions using sensors, the server receives this data and uses the data analysis module to identify the user's behavioral patterns. At the same time, the emotion engine analyzes the data and recognizes that the user is feeling stressed at that time.

[0622] As a result, the server uses machine learning models to predict specific behavioral patterns during the ride and recognizes that the user's current emotional state is stressed. Based on this information, the vehicle's climate control module activates, changing the lighting to softer, warmer colors and playing relaxing music.

[0623] Similarly, if the device collects sensor data while the user is traveling and the emotion engine recognizes that the user is feeling happy, the server can respond by adjusting the settings to brighten the lights and play upbeat music, helping the user feel more comfortable while traveling.

[0624] Prompt Sentence Examples

[0625] "If a user is feeling stressed, how can we adjust the in-car environment to make them more relaxed?"

[0626] "Please explain how to optimize the in-car environment for a happy user."

[0627] In this way, the system can be applied to improve user comfort in vehicles as well.

[0628] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0629] Step 1:

[0630] The terminal collects user behavioral data and emotional data (tone of voice, facial expressions, etc.) in real time from various sensors installed inside the vehicle (temperature sensors, motion sensors, illuminance sensors, voice recognition sensors, etc.). This sensor data becomes the input for the terminal. The input data undergoes initial processing, including preprocessing such as noise removal and format standardization. The terminal then transmits the processed sensor data to a server via the network.

[0631] Step 2:

[0632] The server receives the sensor data sent from the device and analyzes the user's behavioral patterns using a data analysis module. In this analysis, the collected data is treated as variables (time, location, behavior, etc.). An analysis algorithm is used to identify the behavioral patterns, and the specific behavioral patterns are output as the result.

[0633] Step 3:

[0634] The server trains a machine learning model based on the analysis results. The input data is behavioral patterns and associated emotional data. Using this data, the machine learning model learns the user's daily behavioral patterns. Training continues over multiple epochs, and the final output is a model that can predict the user's future behavior.

[0635] Step 4:

[0636] The server runs an emotion engine that uses machine learning models to predict the user's emotional state in real time. Whenever sensor data is input, the emotion engine analyzes the data and outputs the user's current emotional state.

[0637] Step 5:

[0638] The server combines the output of the emotion engine with the predictions of the machine learning model and sends them to the vehicle climate control module. The inputs are the driver's emotional state and predicted behavioral patterns, and the optimal settings for the vehicle's climate are output based on these. These settings are then output as signals that automatically adjust elements such as lighting, air conditioning, and music.

[0639] Step 6:

[0640] The vehicle climate control module automatically adjusts the in-car environment (lighting, air conditioning, music, etc.) based on signals sent from the server. Specifically, it changes the color and brightness of the lights, sets the appropriate temperature, and plays music that matches the user's emotional state, thereby improving user comfort in real time.

[0641] In this way, each step works in coordination to realize a system that provides an optimal in-car environment that matches the emotional state of the user while riding.

[0642] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0643] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0644] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0645] [Third embodiment]

[0646] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0647] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0648] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0649] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0650] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0651] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0652] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0653] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0654] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0655] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0656] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0657] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0658] The present invention is a system that collects and analyzes user sensor data to identify behavioral patterns, uses this data to predict future behavior using a machine learning model, and automatically adjusts the settings of smart home devices based on the prediction results.

[0659] The system consists of the following main components:

[0660] 1. User data collection module

[0661] 2. Data Analysis Module

[0662] 3. Machine Learning Models

[0663] 4. Smart Home Device Control Module

[0664] 1. User data collection module

[0665] The device collects real-time sensor data such as temperature, illuminance, and sound from sensors installed in the room and from the user's smartphone, and then transmits this data to a server via the network.

[0666] 2. Data Analysis Module

[0667] The server passes the collected sensor data to a data analysis module, which uses this data to identify user behavior patterns. For example, if a user is in the living room between 7 and 9 in the morning, this behavior pattern will be identified.

[0668] 3. Machine Learning Models

[0669] The server trains a machine learning model based on the behavioral patterns obtained from the data analysis module. This model can learn from past data and predict future behavior. For example, it can learn that a user wakes up at 7 a.m. every morning and eats breakfast in the living room.

[0670] 4. Smart Home Device Control Module

[0671] The server then operates the smart home device control module based on the prediction results. This module automatically adjusts environmental settings such as lighting, temperature, and music according to whether the user is at home, away from home, or asleep. For example, if the server predicts that the user is at home, it can turn on the lights, set the room temperature to 22 degrees, and play relaxing music.

[0672] Specific examples

[0673] For example, if a device's sensor detects that the user is in the living room at 7:00 AM, the following process occurs: The server receives this data, and the data analysis module identifies the user's behavioral patterns. As a result, it is determined that the user is active in the living room between 7:00 and 9:00 AM. This data is used to train a machine learning model, which predicts that the user will be active in the living room at the same time the next day. Based on this prediction, the server uses the smart home device control module to turn on the lights before 7:00 AM, set the temperature to 22 degrees, and play relaxing music. This allows the user to wake up in a comfortable environment.

[0674] Similarly, when the device detects that the user has left the house, the server will turn off the lights and set the temperature to 18 degrees to save energy. When it's time to go to bed, the server will turn off the lights, set the temperature to 20 degrees, and play sleep-inducing music. In this way, the system can automatically provide the optimal environment tailored to the user's lifestyle.

[0675] The processing flow will be explained below.

[0676] Step 1:

[0677] The device collects user behavior data in real time from various sensors in the home (temperature sensors, motion sensors, light sensors, etc.).

[0678] Step 2:

[0679] The terminals transmit the collected sensor data to a server via a network, where the data is accumulated in real time.

[0680] Step 3:

[0681] The server stores the received sensor data in the user's data collection module, which stores and manages the data.

[0682] Step 4:

[0683] The server analyzes the accumulated data using a data analysis module, which identifies user behavior patterns, such as being in the living room at a specific time.

[0684] Step 5:

[0685] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0686] Step 6:

[0687] As users go about their daily lives, new sensor data is generated, and this data is collected again by the device.

[0688] Step 7:

[0689] The server inputs new data into a machine learning model to predict the user's future behavior, such as whether the user will be at home or out at a certain time the next day.

[0690] Step 8:

[0691] The server then operates a smart home device control module based on the prediction results, which automatically adjusts settings such as lighting, heating, and music.

[0692] Step 9:

[0693] The device receives the new settings from the server and applies them to the smart home devices, thus preparing an optimal environment for the predicted behavior.

[0694] Step 10:

[0695] Users enjoy an optimized home environment: for example, lights are turned on, the temperature is set to 22 degrees, and relaxing music is played when they are at home, and when they leave, the lights are turned off to optimize energy consumption.

[0696] By repeating these steps, the system continuously learns and adapts to the user's lifestyle patterns, achieving both a comfortable and energy-efficient home environment.

[0697] Example 1

[0698] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0699] Conventional smart home systems cannot accurately grasp user behavior patterns and can only set fixed settings for general time periods and situations, which means they do not fully achieve user comfort or energy efficiency.In addition, there is a lack of systems that can automatically collect and analyze sensor data, identify behavior patterns, and predict future behavior as a series of processes, making it difficult to respond to specific user needs.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0701] In this invention, the server includes means for transmitting user sensor data over a network, means for analyzing the received sensor data to identify user behavior patterns, means for training a machine learning model based on the identified behavior patterns to predict the user's future behavior, and means for automatically adjusting the settings of smart home devices based on the prediction results, thereby enabling detailed environmental settings based on user behavior and improving comfort and energy efficiency.

[0702] "Sensor data" refers to information such as temperature, light level, sound, and movement acquired by sensors to detect user behavior and environmental conditions.

[0703] A "network" is a communications infrastructure used by multiple terminals to exchange data with each other.

[0704] "Data analytics" is a technology for processing collected sensor data and identifying user behavior patterns and environmental conditions.

[0705] "Behavioral patterns" refer to the tendency of a user to take certain actions or engage in certain activities at certain times of the day and in certain places.

[0706] A "machine learning model" is an algorithm or program that learns from past data and predicts future behavior and states.

[0707] "Future behavior prediction" is the process of estimating what actions a user will take in the future based on data collected and analyzed in the past.

[0708] "Smart home devices" are electrical and electronic devices for the home that can be remotely or automatically controlled via a network.

[0709] "Automatically adjusting settings" means that the system automatically changes the operating state of smart home devices based on predictions.

[0710] The system of the present invention collects user sensor data, analyzes it to identify behavioral patterns, uses machine learning models to predict future behavior, and automatically adjusts the settings of smart home devices based on those predictions. The main components of the system are:

[0711] 1. User data collection module

[0712] 2. Data Analysis Module

[0713] 3. Machine Learning Models

[0714] 4. Smart Home Device Control Module

[0715] 1. User data collection module

[0716] The device collects real-time sensor data such as temperature, light, sound, and movement from various sensors installed in the room and from the user's smartphone. For example, it can detect and collect data on the temperature, light, and decibel levels of sound in a living room. The device then transmits the collected data to a server via a network. Specific hardware used includes a temperature sensor, light sensor, and microphone.

[0717] 2. Data Analysis Module

[0718] The server receives sensor data sent from the device and passes it to the data analysis module. The data analysis module preprocesses the received data, corrects outliers and missing values, and identifies user behavior patterns. For example, it can identify a pattern in which a user is active in the living room between 7 and 9 every morning. Data analysis tools such as Python and R can be used for the analysis.

[0719] 3. Machine Learning Models

[0720] The server trains a machine learning model based on the behavioral patterns identified by the data analysis module. The machine learning model can learn from past behavioral pattern data and predict future behavior. For example, it can predict whether the user will be in the living room at 7:00 the next day. Machine learning frameworks such as TensorFlow and Scikit-learn can be used for machine learning.

[0721] 4. Smart Home Device Control Module

[0722] The server runs a smart home device control module based on behavioral predictions made by the machine learning model. This module automatically adjusts settings such as lighting, temperature, and music according to the user's predicted behavior. For example, if the module predicts that the user will be in the living room at 7 a.m., it can turn on the lights, set the temperature to 22 degrees, and play relaxing music. Smart home devices include smart lights, smart thermostats, and smart speakers.

[0723] Specific examples

[0724] For example, if a device's sensor detects that the user is in the living room at 7:00 AM, the following process occurs: The server receives this data, and the data analysis module identifies the user's behavioral patterns. As a result, it is determined that the user is active in the living room between 7:00 and 9:00 AM. This data is used to train a machine learning model, which predicts that the user will be active in the living room at the same time the next day. Based on this prediction, the server uses the smart home device control module to turn on the lights before 7:00 AM, set the temperature to 22 degrees, and play relaxing music. This allows the user to wake up in a comfortable environment.

[0725] Examples of prompt statements

[0726] "Please explain how a system can learn a user's behavioral pattern, such as waking up at 7am every morning and eating breakfast in the living room, and automatically adjust the settings of its smart home devices if it predicts the same behavior the next day."

[0727] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0728] Step 1:

[0729] The device collects sensor data such as temperature, illuminance, sound, and movement in real time from sensors installed in the room and from the user's smartphone. Specifically, it detects that the temperature in the living room is 22 degrees, the illuminance is 300 lux, and the sound decibel level is 40 dB. This sensor data is the input. The device acquires this data at regular intervals (for example, every second) and compiles it into a data set.

[0730] Step 2:

[0731] The device sends the collected sensor data to the server via the network. The sent data set includes all acquired sensor data. This transmission is the output. Specifically, the data is sent from the device to the server using an HTTP POST request.

[0732] Step 3:

[0733] The server receives sensor data sent from the device and passes it to the data analysis module. The received data is the input, and the processing of the data passed to the data analysis module is the output. Specifically, the server stores the sensor data in a database, which is then read by the data analysis module.

[0734] Step 4:

[0735] The server's data analysis module preprocesses the received sensor data. The input for preprocessing is raw data, and the output is clean data. Specifically, it corrects outliers and fills in missing values. For example, if the temperature shows an abnormally high value, it will exclude that data point.

[0736] Step 5:

[0737] The server's data analysis module analyzes the preprocessed data and identifies user behavior patterns. The input is clean data, and the output is the identified behavior patterns. Specifically, it identifies that a user is performing a specific activity at a specific time (e.g., being in the living room at 7 a.m.) based on time-series data.

[0738] Step 6:

[0739] The server trains a machine learning model based on the identified behavioral patterns. The input is the behavioral pattern data, and the output is the trained model. Specifically, the server uses past behavioral pattern data to train the model using a machine learning algorithm (e.g., TensorFlow or Scikit-learn).

[0740] Step 7:

[0741] The server uses the trained machine learning model to predict future behavior. The input is the trained model and new sensor data, and the output is the predicted result. Specifically, it predicts the behavior pattern for the next day (e.g., the user will be in the living room from 7:00 to 9:00).

[0742] Step 8:

[0743] The server issues instructions to the smart home device control module based on future behavior predictions. The input is the prediction result, and the output is the device control command. Specifically, it instructs the smart home device to turn on the lights before 7:00, set the temperature to 22 degrees, and play relaxing music.

[0744] This allows users to live their daily lives in a comfortable environment.

[0745] (Application example 1)

[0746] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0747] In conventional virtual stores, it was difficult to efficiently collect and analyze user behavior patterns and optimize product suggestions and layouts in real time based on that data. Furthermore, the lack of automatic adjustments to store design based on predicted user behavior limited improvements to the quality of the user experience. For this reason, there was a need to develop a system that could utilize user behavior data to optimize product suggestions and layouts within a virtual store.

[0748] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0749] In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying user behavior patterns, means for training a machine learning model based on the analysis results and predicting the user's future behavior, and means for adjusting the layout and product proposals of the virtual store based on the prediction results, thereby making it possible to optimize the product proposals and layout of the virtual store in real time based on the user behavior data.

[0750] "User sensor data" refers to information about a user's behavior and environment collected from smart devices and sensor devices.

[0751] A "behavioral pattern" is a series of specific actions that indicate a user's habits or tendencies, derived from analyzing collected user data.

[0752] A "machine learning model" is an algorithm or system that learns from data and predicts users' future behavior and tendencies.

[0753] "Predictive tuning" is the process of automatically changing system or environmental settings based on user behavior predicted by a machine learning model.

[0754] A "virtual store" is a commercial establishment operated on the Internet or in a virtual environment, where users can purchase products or use services online.

[0755] "Product suggestion" is the act of presenting the most suitable product to a user based on user behavioral data and predictive data.

[0756] "Real time" refers to ongoing or near-simultaneous events and implies immediate response without delay.

[0757] "Layout optimization" is the process of changing the product placement and design within a store to make it the most efficient and user-friendly, based on user behavior patterns and predictive data.

[0758] This invention is a system that collects user sensor data, analyzes it to identify user behavior patterns, predicts future behavior using machine learning models, and adjusts the layout and product recommendations of a virtual store based on the predictions. The system consists of the following main components:

[0759] 1. User data collection module:

[0760] Sensor data (such as the time the user looks and the product information viewed) is collected in real time from the smart device (e.g., smart glasses) used by the user. This data is sent to a server via a network.

[0761] 2. Data Analysis Module:

[0762] The server passes the collected sensor data to a data analysis module, which uses this data to identify user behavior patterns, such as when a user gazes at a particular product for a long period of time.

[0763] 3. Machine learning models:

[0764] The server trains a machine learning model (using TensorFlow as an example) based on the behavioral patterns obtained from the data analysis module. This model can learn from past data and predict future behavior. For example, it can learn patterns of users frequently viewing certain products and predict which products they are likely to view next.

[0765] 4. Virtual store layout and product proposal adjustment module:

[0766] The server then adjusts the virtual store's layout and product recommendations based on the prediction results. This module changes product placement and design so that users can experience the optimal products and layout in the virtual store. For example, it prioritizes displaying products that the user is looking at and places related products nearby.

[0767] The specific hardware and software used

[0768] Hardware: Smart glasses

[0769] Software: Machine learning model (TensorFlow), data analysis module (Python)

[0770] Specific examples

[0771] For example, imagine a user wearing smart glasses visits a virtual store. The smart glasses' sensors collect the user's gaze data and send it to a server. The server analyzes this data and recognizes that the user is looking at a particular product for a long time. Based on this behavioral pattern, a machine learning model predicts the user's next move. Based on this prediction, the server adjusts the layout of the virtual store to prioritize displaying related products in the user's field of view.

[0772] Prompt Sentence Examples

[0773] "Predict what product a user is likely to look at next in a section where they look for more than 10 seconds (e.g., the TV section). Also, recommend store layout changes based on that prediction."

[0774] In this way, the present invention allows for real-time optimization of the virtual store environment based on user behavior.

[0775] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0776] Step 1:

[0777] The device collects sensor data from the user through the smart glasses, specifically, information on the time the user looks and the products they look at in real time, and sends this data to a server via a network.

[0778] Input: User gaze data, viewed product information

[0779] Output: Sending sensor data to the server

[0780] Step 2:

[0781] The server passes the received sensor data to a data analysis module, which analyzes the sensor data and identifies user behavior patterns, such as when a user gazes at a particular product for a long period of time.

[0782] Input: Sensor data sent from the device

[0783] Output: User behavior patterns

[0784] Step 3:

[0785] The server trains a machine learning model based on the behavioral patterns obtained from the data analysis module. This model learns from past data and has the ability to predict future behavior, such as which products a user is likely to look at next.

[0786] Input: User behavior patterns

[0787] Output: Updates to the machine learning model and predictions of future behavior

[0788] Step 4:

[0789] The server adjusts the virtual store's layout and product suggestions based on the results of the machine learning model. Specifically, it changes product placement and suggestions based on predicted behavior. For example, it prioritizes displaying products that the user is looking at and places related products nearby.

[0790] Input: Prediction results of machine learning model

[0791] Output: Coordinated virtual store layout and product suggestions

[0792] Step 5:

[0793] The server inputs prompts into the generative AI model to enable advanced product suggestions and store layout adjustments. For example, the server inputs a prompt such as, "Predict the next product that the user is likely to look at in the section where they have looked for more than 10 seconds."

[0794] Input: prompt sentence in generative AI model

[0795] Output: Suggested results from the generative AI model

[0796] Through the above processing steps, the present invention makes it possible to optimize the virtual store environment in real time based on user behavior data.

[0797] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0798] The present invention is a system that uses user sensor data to recognize emotions and automatically adjusts the settings of smart home devices based on those emotions, thereby further improving user comfort.

[0799] The system consists of the following main components:

[0800] 1. User data collection module

[0801] 2. Data Analysis Module

[0802] 3. Machine Learning Models

[0803] 4. Emotion Engine

[0804] 5. Smart Home Device Control Module

[0805] 1. User data collection module

[0806] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors in the home (temperature, motion, light, voice recognition, etc.), and transmits this data to a server via the network.

[0807] 2. Data Analysis Module

[0808] The server passes the collected sensor data to a data analysis module, which uses the data to identify user behavior patterns, such as where the user is at a particular time and what actions they are taking.

[0809] 3. Machine Learning Models

[0810] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0811] 4. Emotion Engine

[0812] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0813] 5. Smart Home Device Control Module

[0814] The server operates a smart home device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module automatically adjusts environmental settings according to the user's emotional state. For example, if the user is feeling stressed, it changes the lights to softer colors and plays relaxing music.

[0815] Specific examples

[0816] For example, if the device detects the user's morning activities and uses sensors to collect their tone of voice and facial expressions, the server receives this data and uses the data analysis module to identify the user's behavioral patterns. At the same time, the emotion engine analyzes the data and recognizes that the user is in a bad mood that morning.

[0817] As a result, the server uses machine learning models to predict that the user has a specific behavioral pattern each morning and recognizes that the user's current emotional state is bad. Based on this information, the smart home device control module activates, changing the lighting to softer, warmer colors and playing relaxing music.

[0818] Similarly, if the device collects sensor data during the user's after-work hours and the emotion engine detects that the user is feeling stressed, the server can respond by adjusting settings to dim the lights and play quiet music, allowing the user to enjoy a relaxing environment as soon as they get home.

[0819] As described above, the present invention, combined with an emotion engine, makes it possible to recognize the user's emotional state and automatically provide a home environment that responds to that state, thereby further improving comfort and optimizing energy efficiency within the home.

[0820] The processing flow will be explained below.

[0821] Step 1:

[0822] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors in the home (temperature sensors, motion sensors, light sensors, voice recognition sensors, etc.).

[0823] Step 2:

[0824] The terminals transmit the collected sensor data to a server via a network, where the data is accumulated in real time.

[0825] Step 3:

[0826] The server stores the received sensor data in the user's data collection module, which stores and manages the data.

[0827] Step 4:

[0828] The server analyzes the accumulated data using a data analysis module, which identifies user behavior patterns, such as being in the living room at a particular time.

[0829] Step 5:

[0830] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0831] Step 6:

[0832] As users go about their daily lives, new sensor data is generated, and this data is collected again by the device.

[0833] Step 7:

[0834] The server inputs new data into a machine learning model to predict the user's future behavior, such as whether the user will be at home or out at a certain time the next day.

[0835] Step 8:

[0836] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0837] Step 9:

[0838] The server operates a smart home device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model, and this module automatically adjusts environmental settings according to the user's emotional state.

[0839] Step 10:

[0840] The device receives new settings from the server and applies them to smart home devices, preparing an optimal environment for predicted behaviors and emotional states.

[0841] Step 11:

[0842] Users enjoy an optimized home environment: for example, if the user is feeling stressed, the lights will change to softer colors and relaxing music will play, and when the user leaves the house, the lights will be turned off to optimize energy consumption.

[0843] By repeating these steps, the system continuously learns and adapts to the user's lifestyle patterns and emotional state, a process that helps to balance comfort and energy efficiency in the home environment.

[0844] Example 2

[0845] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0846] While modern smart home technologies have the ability to automatically adjust a user's home environment, this is primarily based on time of day or preset scenarios. There is currently no system that can adjust the home environment in real time according to the user's emotions and behavior, and this has left the improvement of user comfort and optimization of energy efficiency unsatisfactory.

[0847] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying the user's behavioral pattern, means for training a machine learning model based on the analysis results and predicting the user's future behavior, means for analyzing emotion-related data such as the user's tone of voice and facial expression and recognizing the user's emotional state, and means for automatically adjusting the settings of the smart device based on the recognized user's emotional state and the predicted behavioral pattern. This makes it possible to optimally adjust the home environment in real time based on the user's emotions and behavior.

[0848] "User sensor data" is data related to a user, such as temperature, motion, light level, voice tone, and facial expressions.

[0849] "Analysis means" refers to the technology and devices used to identify user behavior patterns based on collected sensor data.

[0850] A "machine learning model" is a model that learns a user's daily behavioral patterns based on collected and analyzed data and predicts future behavior.

[0851] "Voice tone" is emotion-related data extracted from audio signals, such as voice pitch, intonation, and tempo.

[0852] "Facial expression data" is data that indicates information related to emotions extracted from the facial expression of the user.

[0853] "Emotional state" is the user's mental and emotional state as perceived from the analyzed data.

[0854] "Smart devices" are internet-connected devices and systems used to control and regulate the home environment.

[0855] "Automatic adjustment means" refers to a means for automatically changing or adjusting the settings of a smart device based on the user's emotional state and behavioral patterns.

[0856] The present invention is a system that uses user sensor data to recognize emotions and automatically adjusts smart device settings based on those emotions, further improving user comfort and energy efficiency in the home.

[0857] System Configuration

[0858] The system consists of the following main components:

[0859] 1. User data collection module

[0860] 2. Data Analysis Module

[0861] 3. Machine Learning Models

[0862] 4. Emotion Engine

[0863] 5. Smart Device Control Module

[0864] 1. User data collection module

[0865] The device collects data related to the user's behavior and emotions from various sensors in the home (temperature sensor, motion sensor, light sensor, voice recognition sensor, etc.). For example, a temperature sensor installed in the living room measures the current room temperature, and a motion sensor detects the user's movements. The collected data is sent to a server in real time via the network.

[0866] 2. Data Analysis Module

[0867] The server receives sensor data sent from the device and passes it to the data analysis module. Using the Python Pandas library, the received data is formatted and analyzed to identify user behavior patterns. For example, it analyzes information such as which room a user is in at a specific time and what actions they are taking.

[0868] 3. Machine Learning Models

[0869] The server trains a machine learning model based on the results of the data analysis. This model learns the user's daily behavioral patterns and is able to predict future behavior. Machine learning libraries such as Scikit-learn and TensorFlow are used for training. This allows the server to predict future behavior with high accuracy based on the user's past data.

[0870] 4. Emotion Engine

[0871] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine. The emotion engine then analyzes this data using Google Cloud's emotion analysis API and Amazon Rekognition to recognize the user's emotional state in real time. For example, it uses voice recognition to read stress levels from the user's voice and image processing technology to recognize emotions from facial expressions.

[0872] 5. Smart Device Control Module

[0873] The server operates the smart device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module then configures the environment optimally to reflect the user's emotions. For example, if the server determines that the user is feeling stressed, it can instruct the smart lighting system to change the lighting to a soft, warm color and play relaxing music.

[0874] Specific examples

[0875] Morning Cases

[0876] The device uses sensors to collect data on the user's tone of voice and facial expressions, which are then sent to the server. The server processes this data with a data analysis module and determines that the user is in a bad mood. At the same time, a machine learning model predicts that the user has a specific behavioral pattern each morning. Based on this information, the server can activate the smart device control module to change the lighting to a warmer hue and play relaxing music to soothe the user's bad mood.

[0877] Prompt Sentence Examples

[0878] "This system uses user sensor data to recognize emotions and automatically adjust the settings on a smart device. Please explain the process flow of this system in concrete steps. Also, please explain the specific operations from collecting sensor data to adjusting the environment."

[0879] As described above, this system can recognize the user's emotional state in real time and automatically optimize the home environment based on that information, thereby improving user comfort and optimizing energy efficiency within the home.

[0880] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0881] Step 1: Data collection

[0882] The device uses various sensors in the home (such as temperature, motion, illuminance, and voice recognition sensors) to collect data related to the user's behavior and emotions. Specifically, the temperature sensor in the living room measures the current room temperature, and the motion sensor detects the user's movements. The voice recognition sensor captures the user's tone of voice, and the camera collects facial expression data. This data is sent from the device to the server in real time.

[0883] Input: Sensor data (temperature, movement, light, voice tone, facial expression)

[0884] Output: Raw data sent to the server

[0885] Step 2: Data reception and processing

[0886] The server receives raw data sent from the terminal. The received data is passed to the data analysis module, where preprocessing is performed. Specifically, the Python Pandas library is used to perform tasks such as filling in missing values, removing outliers, and integrating data to ensure consistency across time axes.

[0887] Input: Raw data sent from the terminal

[0888] Output: Preprocessed clean data

[0889] Step 3: Behavioral pattern analysis

[0890] The server analyzes user behavior patterns based on the clean data. The data analysis module identifies which room a user is in at what time of day and what actions they take. For example, it recognizes a pattern that a user is often in the kitchen between 8 and 9 a.m. This analysis is performed using Python's Pandas and NumPy.

[0891] Input: Preprocessed clean data

[0892] Output: User behavior patterns

[0893] Step 4: Train the machine learning model

[0894] The server trains a machine learning model based on the results of the behavioral pattern analysis. Using libraries such as Scikit-learn and TensorFlow, the model is fed with behavioral pattern data to predict future behavior, such as the probability that a user will be in a particular room at a particular time.

[0895] Input: User behavior patterns

[0896] Output: A trained machine learning model

[0897] Step 5: Sentiment Analysis

[0898] The server inputs the user's tone of voice and facial expression data into the emotion engine. This emotion engine uses Google Cloud's emotion analysis API and Amazon Rekognition to analyze the data and recognize the user's emotional state. For example, it can determine stress levels from voice data and identify joy, anger, sadness, and happiness from facial expression data.

[0899] Input: User's voice tone, facial expression data

[0900] Output: User's emotional state

[0901] Step 6: Automatic settings adjustment

[0902] The server operates a smart device control module based on the results of the emotion engine (user's emotional state) and the predictions of the machine learning model (future behavioral patterns). This module automatically adjusts environmental settings based on the user's emotional state and predicted behavior. For example, if the user is feeling stressed, the server instructs the smart lighting system to change the lighting to warmer colors and play relaxing music.

[0903] Input: User's emotional state, prediction results of machine learning model

[0904] Output: Adjusted smart device settings

[0905] (Application example 2)

[0906] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0907] This system solves the problem of improving user comfort by recognizing the user's emotions and behavior in real time while in the car and automatically adjusting the in-car environment based on that. Conventional in-car environment adjustment systems do not take the user's emotions and behavior into account, so they can only provide uniform settings and are unable to provide an optimal environment for each individual user. This means that users have to manually adjust the environment settings, which is inconvenient.

[0908] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0909] In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying the user's behavioral patterns, means for training a machine learning model based on the analysis results and predicting the user's future behavior, and means for automatically adjusting the in-vehicle environmental settings based on the prediction results, thereby making it possible to optimally adjust the in-vehicle environment in real time according to the user's emotions and behavior while riding.

[0910] "Sensor data" refers to data collected from various sensors used to detect user emotions and behavior.

[0911] "Behavioral patterns" indicate the tendencies and habits of a user's behavior over a certain period of time, and are analyzed from sensor data.

[0912] A "machine learning model" is an algorithm that is trained using collected data to predict a user's future behavior.

[0913] "In-vehicle environmental settings" refers to the settings of the vehicle's interior environment (lighting, air conditioning, music, etc.) that are adjusted to improve the user's comfort.

[0914] "At the time of boarding" refers to the time when the user gets into the vehicle.

[0915] "When exiting the vehicle" refers to the point at which the user exits the vehicle.

[0916] "In motion" refers to the time the vehicle is in motion, whether driven or autonomous.

[0917] "Lighting" refers to the brightness and color settings of lights and lighting equipment installed inside the vehicle.

[0918] "Air conditioning" refers to the settings of the equipment used to regulate the temperature and humidity inside the vehicle.

[0919] "Music" refers to songs and sounds played in the vehicle, selected according to the user's emotional state.

[0920] This invention is a system that recognizes the user's emotions and behavior in real time while riding and automatically adjusts the in-car environment based on that. The system includes the following main components:

[0921] 1. User data collection module

[0922] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors installed in the vehicle (temperature sensors, motion sensors, illuminance sensors, voice recognition sensors, etc.), and transmits this data to a server via the network.

[0923] 2. Data Analysis Module

[0924] The server passes the collected sensor data to a data analysis module, which uses the data to identify user behavior patterns, such as where the user is at a particular time and what actions they are taking.

[0925] 3. Machine Learning Models

[0926] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0927] 4. Emotion Engine

[0928] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[0929] 5. Vehicle climate control module

[0930] The server operates the vehicle's environmental control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module automatically adjusts the vehicle's environmental settings (lighting, air conditioning, music, etc.) according to the user's emotional state. For example, if the user is feeling stressed, the module will change the lighting to softer colors and play relaxing music.

[0931] Specific examples

[0932] For example, if the device senses the user's emotions while riding and collects their tone of voice and facial expressions using sensors, the server receives this data and uses the data analysis module to identify the user's behavioral patterns. At the same time, the emotion engine analyzes the data and recognizes that the user is feeling stressed at that time.

[0933] As a result, the server uses machine learning models to predict specific behavioral patterns during the ride and recognizes that the user's current emotional state is stressed. Based on this information, the vehicle's climate control module activates, changing the lighting to softer, warmer colors and playing relaxing music.

[0934] Similarly, if the device collects sensor data while the user is traveling and the emotion engine recognizes that the user is feeling happy, the server can respond by adjusting the settings to brighten the lights and play upbeat music, helping the user feel more comfortable while traveling.

[0935] Prompt Sentence Examples

[0936] "If a user is feeling stressed, how can we adjust the in-car environment to make them more relaxed?"

[0937] "Please explain how to optimize the in-car environment for a happy user."

[0938] In this way, the system can be applied to improve user comfort in vehicles as well.

[0939] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0940] Step 1:

[0941] The terminal collects user behavioral data and emotional data (tone of voice, facial expressions, etc.) in real time from various sensors installed inside the vehicle (temperature sensors, motion sensors, illuminance sensors, voice recognition sensors, etc.). This sensor data becomes the input for the terminal. The input data undergoes initial processing, including preprocessing such as noise removal and format standardization. The terminal then transmits the processed sensor data to a server via the network.

[0942] Step 2:

[0943] The server receives the sensor data sent from the device and analyzes the user's behavioral patterns using a data analysis module. In this analysis, the collected data is treated as variables (time, location, behavior, etc.). An analysis algorithm is used to identify the behavioral patterns, and the specific behavioral patterns are output as the result.

[0944] Step 3:

[0945] The server trains a machine learning model based on the analysis results. The input data is behavioral patterns and associated emotional data. Using this data, the machine learning model learns the user's daily behavioral patterns. Training continues over multiple epochs, and the final output is a model that can predict the user's future behavior.

[0946] Step 4:

[0947] The server runs an emotion engine that uses machine learning models to predict the user's emotional state in real time. Whenever sensor data is input, the emotion engine analyzes the data and outputs the user's current emotional state.

[0948] Step 5:

[0949] The server combines the output of the emotion engine with the predictions of the machine learning model and sends them to the vehicle climate control module. The inputs are the driver's emotional state and predicted behavioral patterns, and the optimal settings for the vehicle's climate are output based on these. These settings are then output as signals that automatically adjust elements such as lighting, air conditioning, and music.

[0950] Step 6:

[0951] The vehicle climate control module automatically adjusts the in-car environment (lighting, air conditioning, music, etc.) based on signals sent from the server. Specifically, it changes the color and brightness of the lights, sets the appropriate temperature, and plays music that matches the user's emotional state, thereby improving user comfort in real time.

[0952] In this way, each step works in coordination to realize a system that provides an optimal in-car environment that matches the emotional state of the user while riding.

[0953] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0954] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0955] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0956] [Fourth embodiment]

[0957] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0958] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0959] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0960] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0961] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0962] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0963] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0964] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0965] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0966] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0967] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0968] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0969] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0970] The present invention is a system that collects and analyzes user sensor data to identify behavioral patterns, uses this data to predict future behavior using a machine learning model, and automatically adjusts the settings of smart home devices based on the prediction results.

[0971] The system consists of the following main components:

[0972] 1. User data collection module

[0973] 2. Data Analysis Module

[0974] 3. Machine Learning Models

[0975] 4. Smart Home Device Control Module

[0976] 1. User data collection module

[0977] The device collects real-time sensor data such as temperature, illuminance, and sound from sensors installed in the room and from the user's smartphone, and then transmits this data to a server via the network.

[0978] 2. Data Analysis Module

[0979] The server passes the collected sensor data to a data analysis module, which uses this data to identify user behavior patterns. For example, if a user is in the living room between 7 and 9 in the morning, this behavior pattern will be identified.

[0980] 3. Machine Learning Models

[0981] The server trains a machine learning model based on the behavioral patterns obtained from the data analysis module. This model can learn from past data and predict future behavior. For example, it can learn that a user wakes up at 7 a.m. every morning and eats breakfast in the living room.

[0982] 4. Smart Home Device Control Module

[0983] The server then operates the smart home device control module based on the prediction results. This module automatically adjusts environmental settings such as lighting, temperature, and music according to whether the user is at home, away from home, or asleep. For example, if the server predicts that the user is at home, it can turn on the lights, set the room temperature to 22 degrees, and play relaxing music.

[0984] Specific examples

[0985] For example, if a device's sensor detects that the user is in the living room at 7:00 AM, the following process occurs: The server receives this data, and the data analysis module identifies the user's behavioral patterns. As a result, it is determined that the user is active in the living room between 7:00 and 9:00 AM. This data is used to train a machine learning model, which predicts that the user will be active in the living room at the same time the next day. Based on this prediction, the server uses the smart home device control module to turn on the lights before 7:00 AM, set the temperature to 22 degrees, and play relaxing music. This allows the user to wake up in a comfortable environment.

[0986] Similarly, when the device detects that the user has left the house, the server will turn off the lights and set the temperature to 18 degrees to save energy. When it's time to go to bed, the server will turn off the lights, set the temperature to 20 degrees, and play sleep-inducing music. In this way, the system can automatically provide the optimal environment tailored to the user's lifestyle.

[0987] The processing flow will be explained below.

[0988] Step 1:

[0989] The device collects user behavior data in real time from various sensors in the home (temperature sensors, motion sensors, light sensors, etc.).

[0990] Step 2:

[0991] The terminals transmit the collected sensor data to a server via a network, where the data is accumulated in real time.

[0992] Step 3:

[0993] The server stores the received sensor data in the user's data collection module, which stores and manages the data.

[0994] Step 4:

[0995] The server analyzes the accumulated data using a data analysis module, which identifies user behavior patterns, such as being in the living room at a specific time.

[0996] Step 5:

[0997] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[0998] Step 6:

[0999] As users go about their daily lives, new sensor data is generated, and this data is collected again by the device.

[1000] Step 7:

[1001] The server inputs new data into a machine learning model to predict the user's future behavior, such as whether the user will be at home or out at a certain time the next day.

[1002] Step 8:

[1003] The server then operates a smart home device control module based on the prediction results, which automatically adjusts settings such as lighting, heating, and music.

[1004] Step 9:

[1005] The device receives the new settings from the server and applies them to the smart home devices, thus preparing an optimal environment for the predicted behavior.

[1006] Step 10:

[1007] Users enjoy an optimized home environment: for example, lights are turned on, the temperature is set to 22 degrees, and relaxing music is played when they are at home, and when they leave, the lights are turned off to optimize energy consumption.

[1008] By repeating these steps, the system continuously learns and adapts to the user's lifestyle patterns, achieving both a comfortable and energy-efficient home environment.

[1009] Example 1

[1010] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1011] Conventional smart home systems cannot accurately grasp user behavior patterns and can only set fixed settings for general time periods and situations, which means they do not fully achieve user comfort or energy efficiency.In addition, there is a lack of systems that can automatically collect and analyze sensor data, identify behavior patterns, and predict future behavior as a series of processes, making it difficult to respond to specific user needs.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1013] In this invention, the server includes means for transmitting user sensor data over a network, means for analyzing the received sensor data to identify user behavior patterns, means for training a machine learning model based on the identified behavior patterns to predict the user's future behavior, and means for automatically adjusting the settings of smart home devices based on the prediction results, thereby enabling detailed environmental settings based on user behavior and improving comfort and energy efficiency.

[1014] "Sensor data" refers to information such as temperature, light level, sound, and movement acquired by sensors to detect user behavior and environmental conditions.

[1015] A "network" is a communications infrastructure used by multiple terminals to exchange data with each other.

[1016] "Data analytics" is a technology for processing collected sensor data and identifying user behavior patterns and environmental conditions.

[1017] "Behavioral patterns" refer to the tendency of a user to take certain actions or engage in certain activities at certain times of the day and in certain places.

[1018] A "machine learning model" is an algorithm or program that learns from past data and predicts future behavior and states.

[1019] "Future behavior prediction" is the process of estimating what actions a user will take in the future based on data collected and analyzed in the past.

[1020] "Smart home devices" are electrical and electronic devices for the home that can be remotely or automatically controlled via a network.

[1021] "Automatically adjusting settings" means that the system automatically changes the operating state of smart home devices based on predictions.

[1022] The system of the present invention collects user sensor data, analyzes it to identify behavioral patterns, uses machine learning models to predict future behavior, and automatically adjusts the settings of smart home devices based on those predictions. The main components of the system are:

[1023] 1. User data collection module

[1024] 2. Data Analysis Module

[1025] 3. Machine Learning Models

[1026] 4. Smart Home Device Control Module

[1027] 1. User data collection module

[1028] The device collects real-time sensor data such as temperature, light, sound, and movement from various sensors installed in the room and from the user's smartphone. For example, it can detect and collect data on the temperature, light, and decibel levels of sound in a living room. The device then transmits the collected data to a server via a network. Specific hardware used includes a temperature sensor, light sensor, and microphone.

[1029] 2. Data Analysis Module

[1030] The server receives sensor data sent from the device and passes it to the data analysis module. The data analysis module preprocesses the received data, corrects outliers and missing values, and identifies user behavior patterns. For example, it can identify a pattern in which a user is active in the living room between 7 and 9 every morning. Data analysis tools such as Python and R can be used for the analysis.

[1031] 3. Machine Learning Models

[1032] The server trains a machine learning model based on the behavioral patterns identified by the data analysis module. The machine learning model can learn from past behavioral pattern data and predict future behavior. For example, it can predict whether the user will be in the living room at 7:00 the next day. Machine learning frameworks such as TensorFlow and Scikit-learn can be used for machine learning.

[1033] 4. Smart Home Device Control Module

[1034] The server runs a smart home device control module based on behavioral predictions made by the machine learning model. This module automatically adjusts settings such as lighting, temperature, and music according to the user's predicted behavior. For example, if the module predicts that the user will be in the living room at 7 a.m., it can turn on the lights, set the temperature to 22 degrees, and play relaxing music. Smart home devices include smart lights, smart thermostats, and smart speakers.

[1035] Specific examples

[1036] For example, if a device's sensor detects that the user is in the living room at 7:00 AM, the following process occurs: The server receives this data, and the data analysis module identifies the user's behavioral patterns. As a result, it is determined that the user is active in the living room between 7:00 and 9:00 AM. This data is used to train a machine learning model, which predicts that the user will be active in the living room at the same time the next day. Based on this prediction, the server uses the smart home device control module to turn on the lights before 7:00 AM, set the temperature to 22 degrees, and play relaxing music. This allows the user to wake up in a comfortable environment.

[1037] Examples of prompt statements

[1038] "Please explain how a system can learn a user's behavioral pattern, such as waking up at 7am every morning and eating breakfast in the living room, and automatically adjust the settings of its smart home devices if it predicts the same behavior the next day."

[1039] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1040] Step 1:

[1041] The device collects sensor data such as temperature, illuminance, sound, and movement in real time from sensors installed in the room and from the user's smartphone. Specifically, it detects that the temperature in the living room is 22 degrees, the illuminance is 300 lux, and the sound decibel level is 40 dB. This sensor data is the input. The device acquires this data at regular intervals (for example, every second) and compiles it into a data set.

[1042] Step 2:

[1043] The device sends the collected sensor data to the server via the network. The sent data set includes all acquired sensor data. This transmission is the output. Specifically, the data is sent from the device to the server using an HTTP POST request.

[1044] Step 3:

[1045] The server receives sensor data sent from the device and passes it to the data analysis module. The received data is the input, and the processing of the data passed to the data analysis module is the output. Specifically, the server stores the sensor data in a database, which is then read by the data analysis module.

[1046] Step 4:

[1047] The server's data analysis module preprocesses the received sensor data. The input for preprocessing is raw data, and the output is clean data. Specifically, it corrects outliers and fills in missing values. For example, if the temperature shows an abnormally high value, it will exclude that data point.

[1048] Step 5:

[1049] The server's data analysis module analyzes the preprocessed data and identifies user behavior patterns. The input is clean data, and the output is the identified behavior patterns. Specifically, it identifies that a user is performing a specific activity at a specific time (e.g., being in the living room at 7 a.m.) based on time-series data.

[1050] Step 6:

[1051] The server trains a machine learning model based on the identified behavioral patterns. The input is the behavioral pattern data, and the output is the trained model. Specifically, the server uses past behavioral pattern data to train the model using a machine learning algorithm (e.g., TensorFlow or Scikit-learn).

[1052] Step 7:

[1053] The server uses the trained machine learning model to predict future behavior. The input is the trained model and new sensor data, and the output is the predicted result. Specifically, it predicts the behavior pattern for the next day (e.g., the user will be in the living room from 7:00 to 9:00).

[1054] Step 8:

[1055] The server issues instructions to the smart home device control module based on future behavior predictions. The input is the prediction result, and the output is the device control command. Specifically, it instructs the smart home device to turn on the lights before 7:00, set the temperature to 22 degrees, and play relaxing music.

[1056] This allows users to live their daily lives in a comfortable environment.

[1057] (Application example 1)

[1058] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1059] In conventional virtual stores, it was difficult to efficiently collect and analyze user behavior patterns and optimize product suggestions and layouts in real time based on that data. Furthermore, the lack of automatic adjustments to store design based on predicted user behavior limited improvements to the quality of the user experience. For this reason, there was a need to develop a system that could utilize user behavior data to optimize product suggestions and layouts within a virtual store.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1061] In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying user behavior patterns, means for training a machine learning model based on the analysis results and predicting the user's future behavior, and means for adjusting the layout and product proposals of the virtual store based on the prediction results, thereby making it possible to optimize the product proposals and layout of the virtual store in real time based on the user behavior data.

[1062] "User sensor data" refers to information about a user's behavior and environment collected from smart devices and sensor devices.

[1063] A "behavioral pattern" is a series of specific actions that indicate a user's habits or tendencies, derived from analyzing collected user data.

[1064] A "machine learning model" is an algorithm or system that learns from data and predicts users' future behavior and tendencies.

[1065] "Predictive tuning" is the process of automatically changing system or environmental settings based on user behavior predicted by a machine learning model.

[1066] A "virtual store" is a commercial establishment operated on the Internet or in a virtual environment, where users can purchase products or use services online.

[1067] "Product suggestion" is the act of presenting the most suitable product to a user based on user behavioral data and predictive data.

[1068] "Real time" refers to ongoing or near-simultaneous events and implies immediate response without delay.

[1069] "Layout optimization" is the process of changing the product placement and design within a store to make it the most efficient and user-friendly, based on user behavior patterns and predictive data.

[1070] This invention is a system that collects user sensor data, analyzes it to identify user behavior patterns, predicts future behavior using machine learning models, and adjusts the layout and product recommendations of a virtual store based on the predictions. The system consists of the following main components:

[1071] 1. User data collection module:

[1072] Sensor data (such as the time the user looks and the product information viewed) is collected in real time from the smart device (e.g., smart glasses) used by the user. This data is sent to a server via a network.

[1073] 2. Data Analysis Module:

[1074] The server passes the collected sensor data to a data analysis module, which uses this data to identify user behavior patterns, such as when a user gazes at a particular product for a long period of time.

[1075] 3. Machine learning models:

[1076] The server trains a machine learning model (using TensorFlow as an example) based on the behavioral patterns obtained from the data analysis module. This model can learn from past data and predict future behavior. For example, it can learn patterns of users frequently viewing certain products and predict which products they are likely to view next.

[1077] 4. Virtual store layout and product proposal adjustment module:

[1078] The server then adjusts the virtual store's layout and product recommendations based on the prediction results. This module changes product placement and design so that users can experience the optimal products and layout in the virtual store. For example, it prioritizes displaying products that the user is looking at and places related products nearby.

[1079] The specific hardware and software used

[1080] Hardware: Smart glasses

[1081] Software: Machine learning model (TensorFlow), data analysis module (Python)

[1082] Specific examples

[1083] For example, imagine a user wearing smart glasses visits a virtual store. The smart glasses' sensors collect the user's gaze data and send it to a server. The server analyzes this data and recognizes that the user is looking at a particular product for a long time. Based on this behavioral pattern, a machine learning model predicts the user's next move. Based on this prediction, the server adjusts the layout of the virtual store to prioritize displaying related products in the user's field of view.

[1084] Prompt Sentence Examples

[1085] "Predict what product a user is likely to look at next in a section where they look for more than 10 seconds (e.g., the TV section). Also, recommend store layout changes based on that prediction."

[1086] In this way, the present invention allows for real-time optimization of the virtual store environment based on user behavior.

[1087] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1088] Step 1:

[1089] The device collects sensor data from the user through the smart glasses, specifically, information on the time the user looks and the products they look at in real time, and sends this data to a server via a network.

[1090] Input: User gaze data, viewed product information

[1091] Output: Sending sensor data to the server

[1092] Step 2:

[1093] The server passes the received sensor data to a data analysis module, which analyzes the sensor data and identifies user behavior patterns, such as when a user gazes at a particular product for a long period of time.

[1094] Input: Sensor data sent from the device

[1095] Output: User behavior patterns

[1096] Step 3:

[1097] The server trains a machine learning model based on the behavioral patterns obtained from the data analysis module. This model learns from past data and has the ability to predict future behavior, such as which products a user is likely to look at next.

[1098] Input: User behavior patterns

[1099] Output: Updates to the machine learning model and predictions of future behavior

[1100] Step 4:

[1101] The server adjusts the virtual store's layout and product suggestions based on the results of the machine learning model. Specifically, it changes product placement and suggestions based on predicted behavior. For example, it prioritizes displaying products that the user is looking at and places related products nearby.

[1102] Input: Prediction results of machine learning model

[1103] Output: Coordinated virtual store layout and product suggestions

[1104] Step 5:

[1105] The server inputs prompts into the generative AI model to enable advanced product suggestions and store layout adjustments. For example, the server inputs a prompt such as, "Predict the next product that the user is likely to look at in the section where they have looked for more than 10 seconds."

[1106] Input: prompt sentence in generative AI model

[1107] Output: Suggested results from the generative AI model

[1108] Through the above processing steps, the present invention makes it possible to optimize the virtual store environment in real time based on user behavior data.

[1109] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1110] The present invention is a system that uses user sensor data to recognize emotions and automatically adjusts the settings of smart home devices based on those emotions, thereby further improving user comfort.

[1111] The system consists of the following main components:

[1112] 1. User data collection module

[1113] 2. Data Analysis Module

[1114] 3. Machine Learning Models

[1115] 4. Emotion Engine

[1116] 5. Smart Home Device Control Module

[1117] 1. User data collection module

[1118] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors in the home (temperature, motion, light, voice recognition, etc.), and transmits this data to a server via the network.

[1119] 2. Data Analysis Module

[1120] The server passes the collected sensor data to a data analysis module, which uses the data to identify user behavior patterns, such as where the user is at a particular time and what actions they are taking.

[1121] 3. Machine Learning Models

[1122] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[1123] 4. Emotion Engine

[1124] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[1125] 5. Smart Home Device Control Module

[1126] The server operates a smart home device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module automatically adjusts environmental settings according to the user's emotional state. For example, if the user is feeling stressed, it changes the lights to softer colors and plays relaxing music.

[1127] Specific examples

[1128] For example, if the device detects the user's morning activities and uses sensors to collect their tone of voice and facial expressions, the server receives this data and uses the data analysis module to identify the user's behavioral patterns. At the same time, the emotion engine analyzes the data and recognizes that the user is in a bad mood that morning.

[1129] As a result, the server uses machine learning models to predict that the user has a specific behavioral pattern each morning and recognizes that the user's current emotional state is bad. Based on this information, the smart home device control module activates, changing the lighting to softer, warmer colors and playing relaxing music.

[1130] Similarly, if the device collects sensor data during the user's after-work hours and the emotion engine detects that the user is feeling stressed, the server can respond by adjusting settings to dim the lights and play quiet music, allowing the user to enjoy a relaxing environment as soon as they get home.

[1131] As described above, the present invention, combined with an emotion engine, makes it possible to recognize the user's emotional state and automatically provide a home environment that responds to that state, thereby further improving comfort and optimizing energy efficiency within the home.

[1132] The processing flow will be explained below.

[1133] Step 1:

[1134] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors in the home (temperature sensors, motion sensors, light sensors, voice recognition sensors, etc.).

[1135] Step 2:

[1136] The terminals transmit the collected sensor data to a server via a network, where the data is accumulated in real time.

[1137] Step 3:

[1138] The server stores the received sensor data in the user's data collection module, which stores and manages the data.

[1139] Step 4:

[1140] The server analyzes the accumulated data using a data analysis module, which identifies user behavior patterns, such as being in the living room at a particular time.

[1141] Step 5:

[1142] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[1143] Step 6:

[1144] As users go about their daily lives, new sensor data is generated, and this data is collected again by the device.

[1145] Step 7:

[1146] The server inputs new data into a machine learning model to predict the user's future behavior, such as whether the user will be at home or out at a certain time the next day.

[1147] Step 8:

[1148] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[1149] Step 9:

[1150] The server operates a smart home device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model, and this module automatically adjusts environmental settings according to the user's emotional state.

[1151] Step 10:

[1152] The device receives new settings from the server and applies them to smart home devices, preparing an optimal environment for predicted behaviors and emotional states.

[1153] Step 11:

[1154] Users enjoy an optimized home environment: for example, if the user is feeling stressed, the lights will change to softer colors and relaxing music will play, and when the user leaves the house, the lights will be turned off to optimize energy consumption.

[1155] By repeating these steps, the system continuously learns and adapts to the user's lifestyle patterns and emotional state, a process that helps to balance comfort and energy efficiency in the home environment.

[1156] Example 2

[1157] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1158] While modern smart home technologies have the ability to automatically adjust a user's home environment, this is primarily based on time of day or preset scenarios. There is currently no system that can adjust the home environment in real time according to the user's emotions and behavior, and this has left the improvement of user comfort and optimization of energy efficiency unsatisfactory.

[1159] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying the user's behavioral pattern, means for training a machine learning model based on the analysis results and predicting the user's future behavior, means for analyzing emotion-related data such as the user's tone of voice and facial expression and recognizing the user's emotional state, and means for automatically adjusting the settings of the smart device based on the recognized user's emotional state and the predicted behavioral pattern. This makes it possible to optimally adjust the home environment in real time based on the user's emotions and behavior.

[1160] "User sensor data" is data related to a user, such as temperature, motion, light level, voice tone, and facial expressions.

[1161] "Analysis means" refers to the technology and devices used to identify user behavior patterns based on collected sensor data.

[1162] A "machine learning model" is a model that learns a user's daily behavioral patterns based on collected and analyzed data and predicts future behavior.

[1163] "Voice tone" is emotion-related data extracted from audio signals, such as voice pitch, intonation, and tempo.

[1164] "Facial expression data" is data that indicates information related to emotions extracted from the facial expression of the user.

[1165] "Emotional state" is the user's mental and emotional state as perceived from the analyzed data.

[1166] "Smart devices" are internet-connected devices and systems used to control and regulate the home environment.

[1167] "Automatic adjustment means" refers to a means for automatically changing or adjusting the settings of a smart device based on the user's emotional state and behavioral patterns.

[1168] The present invention is a system that uses user sensor data to recognize emotions and automatically adjusts smart device settings based on those emotions, further improving user comfort and energy efficiency in the home.

[1169] System Configuration

[1170] The system consists of the following main components:

[1171] 1. User data collection module

[1172] 2. Data Analysis Module

[1173] 3. Machine Learning Models

[1174] 4. Emotion Engine

[1175] 5. Smart Device Control Module

[1176] 1. User data collection module

[1177] The device collects data related to the user's behavior and emotions from various sensors in the home (temperature sensor, motion sensor, light sensor, voice recognition sensor, etc.). For example, a temperature sensor installed in the living room measures the current room temperature, and a motion sensor detects the user's movements. The collected data is sent to a server in real time via the network.

[1178] 2. Data Analysis Module

[1179] The server receives sensor data sent from the device and passes it to the data analysis module. Using the Python Pandas library, the received data is formatted and analyzed to identify user behavior patterns. For example, it analyzes information such as which room a user is in at a specific time and what actions they are taking.

[1180] 3. Machine Learning Models

[1181] The server trains a machine learning model based on the results of the data analysis. This model learns the user's daily behavioral patterns and is able to predict future behavior. Machine learning libraries such as Scikit-learn and TensorFlow are used for training. This allows the server to predict future behavior with high accuracy based on the user's past data.

[1182] 4. Emotion Engine

[1183] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine. The emotion engine then analyzes this data using Google Cloud's emotion analysis API and Amazon Rekognition to recognize the user's emotional state in real time. For example, it uses voice recognition to read stress levels from the user's voice and image processing technology to recognize emotions from facial expressions.

[1184] 5. Smart Device Control Module

[1185] The server operates the smart device control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module then configures the environment optimally to reflect the user's emotions. For example, if the server determines that the user is feeling stressed, it can instruct the smart lighting system to change the lighting to a soft, warm color and play relaxing music.

[1186] Specific examples

[1187] Morning Cases

[1188] The device uses sensors to collect data on the user's tone of voice and facial expressions, which are then sent to the server. The server processes this data with a data analysis module and determines that the user is in a bad mood. At the same time, a machine learning model predicts that the user has a specific behavioral pattern each morning. Based on this information, the server can activate the smart device control module to change the lighting to a warmer hue and play relaxing music to soothe the user's bad mood.

[1189] Prompt Sentence Examples

[1190] "This system uses user sensor data to recognize emotions and automatically adjust the settings on a smart device. Please explain the process flow of this system in concrete steps. Also, please explain the specific operations from collecting sensor data to adjusting the environment."

[1191] As described above, this system can recognize the user's emotional state in real time and automatically optimize the home environment based on that information, thereby improving user comfort and optimizing energy efficiency within the home.

[1192] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1193] Step 1: Data collection

[1194] The device uses various sensors in the home (such as temperature, motion, illuminance, and voice recognition sensors) to collect data related to the user's behavior and emotions. Specifically, the temperature sensor in the living room measures the current room temperature, and the motion sensor detects the user's movements. The voice recognition sensor captures the user's tone of voice, and the camera collects facial expression data. This data is sent from the device to the server in real time.

[1195] Input: Sensor data (temperature, movement, light, voice tone, facial expression)

[1196] Output: Raw data sent to the server

[1197] Step 2: Data reception and processing

[1198] The server receives raw data sent from the terminal. The received data is passed to the data analysis module, where preprocessing is performed. Specifically, the Python Pandas library is used to perform tasks such as filling in missing values, removing outliers, and integrating data to ensure consistency across time axes.

[1199] Input: Raw data sent from the terminal

[1200] Output: Preprocessed clean data

[1201] Step 3: Behavioral pattern analysis

[1202] The server analyzes user behavior patterns based on the clean data. The data analysis module identifies which room a user is in at what time of day and what actions they take. For example, it recognizes a pattern that a user is often in the kitchen between 8 and 9 a.m. This analysis is performed using Python's Pandas and NumPy.

[1203] Input: Preprocessed clean data

[1204] Output: User behavior patterns

[1205] Step 4: Train the machine learning model

[1206] The server trains a machine learning model based on the results of the behavioral pattern analysis. Using libraries such as Scikit-learn and TensorFlow, the model is fed with behavioral pattern data to predict future behavior, such as the probability that a user will be in a particular room at a particular time.

[1207] Input: User behavior patterns

[1208] Output: A trained machine learning model

[1209] Step 5: Sentiment Analysis

[1210] The server inputs the user's tone of voice and facial expression data into the emotion engine. This emotion engine uses Google Cloud's emotion analysis API and Amazon Rekognition to analyze the data and recognize the user's emotional state. For example, it can determine stress levels from voice data and identify joy, anger, sadness, and happiness from facial expression data.

[1211] Input: User's voice tone, facial expression data

[1212] Output: User's emotional state

[1213] Step 6: Automatic settings adjustment

[1214] The server operates a smart device control module based on the results of the emotion engine (user's emotional state) and the predictions of the machine learning model (future behavioral patterns). This module automatically adjusts environmental settings based on the user's emotional state and predicted behavior. For example, if the user is feeling stressed, the server instructs the smart lighting system to change the lighting to warmer colors and play relaxing music.

[1215] Input: User's emotional state, prediction results of machine learning model

[1216] Output: Adjusted smart device settings

[1217] (Application example 2)

[1218] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1219] This system solves the problem of improving user comfort by recognizing the user's emotions and behavior in real time while in the car and automatically adjusting the in-car environment based on that. Conventional in-car environment adjustment systems do not take the user's emotions and behavior into account, so they can only provide uniform settings and are unable to provide an optimal environment for each individual user. This means that users have to manually adjust the environment settings, which is inconvenient.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1221] In this invention, the server includes means for collecting user sensor data, means for analyzing the collected user sensor data and identifying the user's behavioral patterns, means for training a machine learning model based on the analysis results and predicting the user's future behavior, and means for automatically adjusting the in-vehicle environmental settings based on the prediction results, thereby making it possible to optimally adjust the in-vehicle environment in real time according to the user's emotions and behavior while riding.

[1222] "Sensor data" refers to data collected from various sensors used to detect user emotions and behavior.

[1223] "Behavioral patterns" indicate the tendencies and habits of a user's behavior over a certain period of time, and are analyzed from sensor data.

[1224] A "machine learning model" is an algorithm that is trained using collected data to predict a user's future behavior.

[1225] "In-vehicle environmental settings" refers to the settings of the vehicle's interior environment (lighting, air conditioning, music, etc.) that are adjusted to improve the user's comfort.

[1226] "At the time of boarding" refers to the time when the user gets into the vehicle.

[1227] "When exiting the vehicle" refers to the point at which the user exits the vehicle.

[1228] "In motion" refers to the time the vehicle is in motion, whether driven or autonomous.

[1229] "Lighting" refers to the brightness and color settings of lights and lighting equipment installed inside the vehicle.

[1230] "Air conditioning" refers to the settings of the equipment used to regulate the temperature and humidity inside the vehicle.

[1231] "Music" refers to songs and sounds played in the vehicle, selected according to the user's emotional state.

[1232] This invention is a system that recognizes the user's emotions and behavior in real time while riding and automatically adjusts the in-car environment based on that. The system includes the following main components:

[1233] 1. User data collection module

[1234] The device collects user behavior data and emotional data such as tone of voice and facial expressions in real time from various sensors installed in the vehicle (temperature sensors, motion sensors, illuminance sensors, voice recognition sensors, etc.), and transmits this data to a server via the network.

[1235] 2. Data Analysis Module

[1236] The server passes the collected sensor data to a data analysis module, which uses the data to identify user behavior patterns, such as where the user is at a particular time and what actions they are taking.

[1237] 3. Machine Learning Models

[1238] The server uses the data analysis results to train a machine learning model that can learn the user's daily behavior patterns and predict future behavior.

[1239] 4. Emotion Engine

[1240] The server inputs emotion-related data, such as the user's tone of voice and facial expressions, into the emotion engine, which then analyzes this data to recognize the user's emotional state, for example, determining whether the user is stressed or relaxed.

[1241] 5. Vehicle climate control module

[1242] The server operates the vehicle's environmental control module based on the user's emotional state recognized by the emotion engine and the behavior predicted by the machine learning model. This module automatically adjusts the vehicle's environmental settings (lighting, air conditioning, music, etc.) according to the user's emotional state. For example, if the user is feeling stressed, the module will change the lighting to softer colors and play relaxing music.

[1243] Specific examples

[1244] For example, if the device senses the user's emotions while riding and collects their tone of voice and facial expressions using sensors, the server receives this data and uses the data analysis module to identify the user's behavioral patterns. At the same time, the emotion engine analyzes the data and recognizes that the user is feeling stressed at that time.

[1245] As a result, the server uses machine learning models to predict specific behavioral patterns during the ride and recognizes that the user's current emotional state is stressed. Based on this information, the vehicle's climate control module activates, changing the lighting to softer, warmer colors and playing relaxing music.

[1246] Similarly, if the device collects sensor data while the user is traveling and the emotion engine recognizes that the user is feeling happy, the server can respond by adjusting the settings to brighten the lights and play upbeat music, helping the user feel more comfortable while traveling.

[1247] Prompt Sentence Examples

[1248] "If a user is feeling stressed, how can we adjust the in-car environment to make them more relaxed?"

[1249] "Please explain how to optimize the in-car environment for a happy user."

[1250] In this way, the system can be applied to improve user comfort in vehicles as well.

[1251] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1252] Step 1:

[1253] The terminal collects user behavioral data and emotional data (tone of voice, facial expressions, etc.) in real time from various sensors installed inside the vehicle (temperature sensors, motion sensors, illuminance sensors, voice recognition sensors, etc.). This sensor data becomes the input for the terminal. The input data undergoes initial processing, including preprocessing such as noise removal and format standardization. The terminal then transmits the processed sensor data to a server via the network.

[1254] Step 2:

[1255] The server receives the sensor data sent from the device and analyzes the user's behavioral patterns using a data analysis module. In this analysis, the collected data is treated as variables (time, location, behavior, etc.). An analysis algorithm is used to identify the behavioral patterns, and the specific behavioral patterns are output as the result.

[1256] Step 3:

[1257] The server trains a machine learning model based on the analysis results. The input data is behavioral patterns and associated emotional data. Using this data, the machine learning model learns the user's daily behavioral patterns. Training continues over multiple epochs, and the final output is a model that can predict the user's future behavior.

[1258] Step 4:

[1259] The server runs an emotion engine that uses machine learning models to predict the user's emotional state in real time. Whenever sensor data is input, the emotion engine analyzes the data and outputs the user's current emotional state.

[1260] Step 5:

[1261] The server combines the output of the emotion engine with the predictions of the machine learning model and sends them to the vehicle climate control module. The inputs are the driver's emotional state and predicted behavioral patterns, and the optimal settings for the vehicle's climate are output based on these. These settings are then output as signals that automatically adjust elements such as lighting, air conditioning, and music.

[1262] Step 6:

[1263] The vehicle climate control module automatically adjusts the in-car environment (lighting, air conditioning, music, etc.) based on signals sent from the server. Specifically, it changes the color and brightness of the lights, sets the appropriate temperature, and plays music that matches the user's emotional state, thereby improving user comfort in real time.

[1264] In this way, each step works in coordination to realize a system that provides an optimal in-car environment that matches the emotional state of the user while riding.

[1265] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1266] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1267] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1268] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1269] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1270] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1271] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1272] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1273] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1274] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1275] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1276] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1277] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1278] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1279] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1280] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1281] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1282] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1283] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1284] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1285] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1286] The following is further disclosed regarding the above embodiment.

[1287] (Claim 1)

[1288] a means for collecting user sensor data;

[1289] A means for analyzing the collected user sensor data and identifying user behavior patterns;

[1290] A means to train machine learning models based on the analysis results and predict future user behavior;

[1291] means for automatically adjusting settings of the smart home device based on the prediction results;

[1292] A system including:

[1293] (Claim 2)

[1294] 2. The system according to claim 1, wherein the home environment settings are automatically adjusted in different patterns for each of the following situations: when the user is at home, when the user is out, and when the user is sleeping.

[1295] (Claim 3)

[1296] 10. The system of claim 1, wherein the system adjusts lighting, temperature, and music settings to enhance the comfort of the home environment.

[1297] "Example 1"

[1298] (Claim 1)

[1299] a means for collecting user sensor data;

[1300] a means for transmitting the collected user sensor data over a network;

[1301] means for analyzing the received sensor data and identifying user behavior patterns;

[1302] A means to train a machine learning model based on identified behavioral patterns to predict future user behavior; and

[1303] means for automatically adjusting settings of the smart home device based on the prediction results;

[1304] A system including:

[1305] (Claim 2)

[1306] 2. The system according to claim 1, wherein the home environment settings are automatically adjusted in different patterns for each of the following situations: when the user is at home, when the user is out, and when the user is sleeping.

[1307] (Claim 3)

[1308] 10. The system of claim 1, wherein the system adjusts lighting, temperature, and music settings to enhance the comfort of the home environment.

[1309] "Application Example 1"

[1310] (Claim 1)

[1311] a means for collecting user sensor data;

[1312] A means for analyzing the collected user sensor data and identifying user behavior patterns;

[1313] A means to train machine learning models based on the analysis results and predict future user behavior;

[1314] A means for adjusting the layout and product suggestions of the virtual store based on the prediction results;

[1315] A system including:

[1316] (Claim 2)

[1317] The system of claim 1, wherein the system optimizes the location of products in the virtual store based on user behavior patterns.

[1318] (Claim 3)

[1319] The system of claim 1 converts collected user sensor data into prompt sentences and uses a generative AI model to realize advanced product suggestions and store layout adjustments.

[1320] "Example 2: Combining Emotion Engines"

[1321] (Claim 1)

[1322] a means for collecting user sensor data;

[1323] A means for analyzing the collected user sensor data and identifying user behavior patterns;

[1324] A means to train machine learning models based on the analysis results and predict future user behavior;

[1325] A means for recognizing the emotional state of a user by analyzing emotion-related data such as the user's tone of voice and facial expression;

[1326] means for automatically adjusting settings on the smart device based on the recognized emotional state and predicted behavioral patterns of the user;

[1327] A system including:

[1328] (Claim 2)

[1329] 2. The system according to claim 1, wherein the home environment settings are automatically adjusted in different patterns for each of the following situations: when the user is at home, when the user is out, and when the user is sleeping.

[1330] (Claim 3)

[1331] 10. The system of claim 1, wherein the system adjusts lighting, temperature, and music settings to enhance the comfort of the home environment.

[1332] "Application example 2 when combining emotion engines"

[1333] (Claim 1)

[1334] a means for collecting user sensor data;

[1335] A means for analyzing the collected user sensor data and identifying user behavior patterns;

[1336] A means to train machine learning models based on the analysis results and predict future user behavior;

[1337] means for automatically adjusting environmental settings within the vehicle based on the predicted results;

[1338] A system including:

[1339] (Claim 2)

[1340] The system according to claim 1, wherein the in-vehicle environmental settings are automatically adjusted in different patterns for each scene: when the user gets on and off the vehicle, and while traveling.

[1341] (Claim 3)

[1342] 10. The system of claim 1, wherein the system adjusts lighting, climate control, and music settings to enhance comfort within the vehicle. [Explanation of symbols]

[1343] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting user sensor data; A means for analyzing the collected user sensor data and identifying user behavior patterns; A means to train machine learning models based on the analysis results and predict future user behavior; means for automatically adjusting settings of the smart home device based on the prediction results; A system including:

2. The system according to claim 1, wherein the home environment settings are automatically adjusted in different patterns for each of the scenes when the user is at home, when the user is out, and when the user is sleeping.

3. 10. The system of claim 1, wherein the system adjusts lighting, temperature, and music settings to enhance the comfort of the home environment.

Images