System

Abstract

A system is provided.SOLUTION: A system comprising: means for automatically adjusting lighting, temperature, music, and video based on user behavior patterns and preferences; means for analyzing the behavior patterns and outside temperature and predictively adjusting appropriate room temperature; and means for providing interactive entertainment in response to user responses and selections.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional living spaces, it has been difficult to adjust the environment to suit the user's lifestyle and individual needs. Furthermore, there has been no system that provides convenience and comfort in a wide range of areas, such as entertainment and remote work, all in one place. Furthermore, it has been difficult to introduce a system that supports health while optimizing energy efficiency. [Means for solving the problem]

[0005] The present invention provides a system that automatically adjusts lighting, temperature, music, and video based on a user's behavioral patterns and preferences, predictively adjusts room temperature by analyzing behavioral patterns and outdoor temperatures, and provides interactive entertainment based on the user's responses and selections. This allows the system to constantly maintain an environment optimal for the user's lifestyle, balancing comfort and convenience. The system also includes an energy efficiency optimization mechanism to minimize energy consumption and a health support mechanism that analyzes sleep patterns and activity levels. Furthermore, the system can recognize the user's movements and provide interactive exercise and entertainment. The system also includes a mechanism that links lighting, temperature, and music to create an environment for relaxation and improved concentration, and a mechanism that supports remote meetings by providing a large screen for video conferencing.

[0006] A "behavioral pattern" is a series of data that indicates a user's daily actions and habits.

[0007] "Preferences" is data that indicates a user's tendency or taste to prefer a particular environment or setting.

[0008] "Automatically adjust" means that the system automatically changes the settings without user intervention.

[0009] "Outside temperature" refers to the temperature outside the living space, and is used as reference data when the system adjusts the environment.

[0010] "Predictive adjustment" refers to predicting future conditions based on collected data and adjusting settings in advance.

[0011] "Interactive entertainment" is a form of entertainment in which the user reacts in real time and the system dynamically adjusts the content based on that reaction.

[0012] "Energy efficiency" refers to providing maximum performance and comfort while minimizing energy consumption.

[0013] "Sleep pattern" is data that indicates information such as the user's sleep cycle, quality, and time period.

[0014] "Activity level" is data indicating the intensity and frequency of the user's daily activities.

[0015] "Exercise" refers to physical activity aimed at moving the body.

[0016] "Relaxation" refers to environments and activities that allow users to relax and reduce stress.

[0017] "Improving concentration" refers to an environment or behavior that allows a user to maintain high levels of attention and work efficiently.

[0018] "Large screens for video conferencing" refers to large displays used for video calls and meetings.

[0019] A "remote meeting" refers to a meeting held over the Internet by multiple people in remote locations. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention relates to a system for making a user's living space more interactive and comfortable. Specific embodiments of the present invention will be described below.

[0042] System configuration

[0043] The system includes the following main elements:

[0044] 1. Server: A central control unit that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[0045] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[0046] 3. Lighting, thermostats, smart speakers, etc.: Devices for adjusting the physical environment, including integrated devices such as the "Smart Ceiling Nexus."

[0047] System Operation

[0048] Preparing the interface

[0049] When the system starts up, the server connects to "SmartCeiling Nexus" and obtains device status and configuration information. The device (smartphone or tablet) connects to the server via the application, and the user profile is loaded. This prepares the system to configure settings based on the user's individual information.

[0050] Providing a personalized experience

[0051] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes the user profile and determines appropriate settings based on the results, which are then sent to the device and "SmartCeiling Nexus."

[0052] Example: If a user prefers dim lighting when watching movies, the server can automatically dim the lighting when the movie is playing.

[0053] Predictive Comfort Management

[0054] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[0055] Example: If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[0056] Interactive Entertainment

[0057] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[0058] Example: Automatically set the lighting to red and the sound effects to emphasize the climax scene of a horror movie.

[0059] Optimizing energy efficiency

[0060] The server monitors the usage of each device and optimizes energy consumption by issuing instructions to the SmartCeiling Nexus to turn off unnecessary devices and functions, minimizing energy consumption.

[0061] Example: Turning off the lights automatically after a certain time has passed after the user leaves the room.

[0062] Health and Wellness Support

[0063] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and configures it to provide an appropriate environment.

[0064] Example: Automatically change the lighting to warmer colors and adjust the temperature before the user goes to sleep.

[0065] Interactive wall / ceiling games

[0066] The server analyzes the camera data, recognizes the user's movements, and sends interactive game content in real time to the "SmartCeiling Nexus," which projects it onto the walls and ceiling.

[0067] Example: When a user plays a jogging game, the course is projected onto a wall and the environment changes as the user moves.

[0068] Environment-based visual experiences

[0069] The server obtains data on the video content being played on the device, determines the optimal environmental settings (lighting, temperature, music), and issues instructions to "SmartCeiling Nexus."

[0070] Example: When playing a yoga video, adjust the lighting to a softer color and set the temperature appropriately to enhance the relaxation effect.

[0071] Optimizing remote work

[0072] The server analyzes the user's remote meeting schedule and provides the optimal environment settings. "SmartCeiling Nexus" starts the projector and adjusts the lighting and sound for the remote meeting.

[0073] Example: Before a remote meeting begins, start the projector and set the lighting appropriately to ensure a smooth start to the meeting.

[0074] These functions allow users to enjoy a comfortable and efficient living environment. The system of the present invention adjusts the environment interactively and automatically to provide an optimal environment according to the user's needs.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] The server connects to the SmartCeiling Nexus when the system starts up and obtains device status and configuration information.

[0078] Step 2:

[0079] The terminal starts a dedicated application and transmits user authentication information to the server.

[0080] Step 3:

[0081] After verifying the authentication information, the server reads the user profile and analyzes past behavioral patterns and preference data.

[0082] Step 4:

[0083] The server determines personalized settings such as lighting, temperature, music, and video based on the analysis results.

[0084] Step 5:

[0085] The setting information determined by the server is sent to the terminal and "SmartCeiling Nexus."

[0086] Step 6:

[0087] "SmartCeiling Nexus" configures the physical environment, such as adjusting the color temperature of lighting, based on the setting information it receives.

[0088] Step 7:

[0089] The server collects real-time external weather data and room usage patterns to predictively determine a comfortable room temperature.

[0090] Step 8:

[0091] Based on instructions from the server, "SmartCeiling Nexus" automatically adjusts the heating and cooling settings.

[0092] Step 9:

[0093] When entertainment content is played on the terminal, the playback information is sent to the server.

[0094] Step 10:

[0095] The server sends instructions to the SmartCeiling Nexus to dynamically change lighting and sound settings based on the type of content and user preferences.

[0096] Step 11:

[0097] The SmartCeiling Nexus adjusts lighting and sound to provide an interactive entertainment experience.

[0098] Step 12:

[0099] The server monitors the energy usage of each device and issues instructions to turn off unnecessary devices.

[0100] Step 13:

[0101] "SmartCeiling Nexus" follows instructions from the server and automatically changes settings to optimize energy efficiency (e.g., turning off lights that have not been used for a long time).

[0102] Step 14:

[0103] The user's sleep and activity data collected on the device is sent to a server.

[0104] Step 15:

[0105] The server analyzes this data and determines settings (e.g., lighting, temperature) to provide an optimal sleeping environment.

[0106] Step 16:

[0107] The server sends instructions to the SmartCeiling Nexus, which adjusts the color temperature of the lighting and room temperature to create an optimal sleeping environment.

[0108] Step 17:

[0109] The server detects the start of an interactive game and analyzes real-time video data from the camera.

[0110] Step 18:

[0111] Game content is sent from the server to the "SmartCeiling Nexus" in real time and projected onto the walls and ceiling.

[0112] Step 19:

[0113] Data of the video content to be played on the terminal is transmitted to the server.

[0114] Step 20:

[0115] The server determines the optimal environmental settings for the video and sends that setting information to "SmartCeiling Nexus."

[0116] Step 21:

[0117] SmartCeiling Nexus adjusts lighting, temperature and music based on user instructions to provide an environment for relaxation and improved concentration.

[0118] Step 22:

[0119] The server analyzes the user's calendar information and checks for remote meeting schedules.

[0120] Step 23:

[0121] The server determines the optimal environment settings for the remote meeting and sends instructions to the SmartCeiling Nexus.

[0122] Step 24:

[0123] SmartCeiling Nexus activates the projector and adjusts lighting and sound for remote meetings.

[0124] Each of these processing steps taken allows the system to provide an interactive and comfortable living space for the user.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] Modern living environments require multiple devices and systems to enhance user comfort and efficiency, but these often operate independently, making integrated management difficult. Furthermore, meeting a wide range of needs, such as optimizing energy consumption, supporting health, and providing interactive entertainment, requires a high degree of automation and precise configuration based on user profiles. Conventional systems struggle to meet these requirements and are unable to provide a fully comfortable living environment for users.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, means for acquiring device status and setting information at system startup, means for reading a user profile and setting the settings based on the individual information, and means for monitoring the user's device usage and optimizing energy efficiency to minimize energy consumption at all times. This makes it possible to manage and improve the user's comfort, health, energy efficiency, and entertainment experience in an integrated manner.

[0130] A "behavioral pattern" refers to the actions or choices a user makes repeatedly in a particular environment or situation.

[0131] "Preferences" refer to the preferences or tendencies of a user that are particularly favored in a particular setting or environment.

[0132] "Lighting" refers to artificial light sources in a user's living space.

[0133] "Temperature" refers to the indoor temperature in the user's living space.

[0134] "Music" refers to audio content that is played in the user's living space.

[0135] "Video" refers to video and image content that is visually displayed in the user's living space.

[0136] "Outside temperature" refers to the temperature of the air outside the living space.

[0137] "Predictive adjustment" means predicting future changes based on past data and current conditions, and making appropriate settings in advance.

[0138] "Reaction" refers to the behavior or response a user exhibits in response to a particular stimulus or situation.

[0139] "Selection" refers to a user selecting a specific item or setting from multiple options.

[0140] "Interactive entertainment" refers to entertainment content that changes dynamically in response to user input and actions.

[0141] "Boot up" refers to getting a system or device up and running and ready for use.

[0142] "Device" refers to electronic devices and machines used in the user's living space.

[0143] "Configuration information" refers to the operating conditions and parameters of a device or system.

[0144] A "user profile" refers to a data set that includes a user's personal information, preferences, behavioral patterns, etc.

[0145] "Individual information" refers to information or data that is unique to a particular user.

[0146] "Monitoring" refers to the continuous watching and recording of the operating status of a particular device or system.

[0147] "Energy efficiency" refers to the efficiency with which a particular device or system uses the energy it consumes.

[0148] The present invention relates to a system that makes a user's living space more interactive and comfortable. This system is mainly composed of a server, a terminal, and various devices (lighting, temperature control devices, smart speakers, etc.). An embodiment of the present invention will now be described in detail.

[0149] System configuration

[0150] The system includes the following main elements:

[0151] 1. Server: A central control unit that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[0152] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[0153] 3. Various devices: Devices for adjusting the physical environment, including "integrated devices" and "sensor clusters."

[0154] System Operation

[0155] Preparing the interface

[0156] When the system starts up, the server connects to the "integrated devices" and obtains the status and setting information of each device. This determines the initial state of the entire system. The device (smartphone or tablet) connects to the server via an application, and the user profile is loaded. This prepares the system to perform settings based on the user's individual information.

[0157] Example: When the system starts up, it receives information about the current room temperature and lighting settings from the integrated device. When you start the app on your smartphone and log in, the server retrieves the user's past activity log and settings.

[0158] Providing a personalized experience

[0159] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes the user profile and determines appropriate settings based on the results, which are then sent to the terminal and each device.

[0160] Example: When a user returns home, the living room lights automatically change to a warmer color and the settings are changed to play their favorite music.

[0161] Predictive Comfort Management

[0162] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[0163] Example: If the outside temperature is predicted to be high, the air conditioner will automatically turn on and adjust the set temperature appropriately.

[0164] Device usage monitoring and energy management

[0165] The server monitors the usage of each device and optimizes energy consumption, issuing instructions to turn off unnecessary devices and functions.

[0166] Example: Lights turn off automatically after a certain time has passed since the last time the user operated them.

[0167] Health data analysis and response

[0168] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and sets optimal settings.

[0169] Example: When the user goes to sleep, the lights change to warm colors to automatically create a comfortable sleeping environment.

[0170] Executing interactive functions

[0171] The server analyzes the camera data, recognizes the user's movements, and sends interactive game content in real time to the "integrated device," which projects it onto the walls and ceiling.

[0172] Example: Dynamic graphics are projected onto the wall as a user plays an exercise game with their family.

[0173] Preferences based on video content

[0174] The server obtains data on the video content being played on the terminal, determines the optimal environment settings, and issues instructions to each device.

[0175] Example: When you play a yoga video, the lighting changes to a softer, more relaxing color and the room temperature adjusts appropriately.

[0176] Optimizing the remote work environment

[0177] The server analyzes the user's remote conference schedule and provides optimal environment settings, while the integrated device activates the projector and adjusts the lighting and sound for the remote conference.

[0178] Example: The projector automatically starts 15 minutes before the start of a meeting, and the brightness and sound settings are adjusted to suit the meeting.

[0179] With these functions, users can enjoy a comfortable and efficient living environment. The system adjusts the environment interactively and automatically to provide an optimal environment according to the user's needs.

[0180] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0181] Step 1: Boot up the system and connect devices

[0182] When the system starts up, the server connects to the integrated devices and obtains the status and setting information of each device. This determines the initial state of the entire system. The input is the system startup signal and the response from the integrated devices, and the output is the current status and setting information of each device.

[0183] How it works: When the system starts up, the server sends a signal to the integrated device and receives information about the current room temperature and lighting settings.

[0184] Step 2: Loading the User Profile

[0185] The device (smartphone or tablet) connects to the server via an application and reads the user profile. The input is the device's login information, and the output is profile information that includes the user's past behavioral data and preferences.

[0186] Specific operation: When a user launches the app on their smartphone and logs in, the server retrieves the user's past activity logs and settings.

[0187] Step 3: Apply personalized settings

[0188] The server analyzes the user profile and automatically adjusts lighting, temperature, music, video, etc. based on past behavioral patterns and preferences. The input is the user profile and current environmental data, and the output is automatically adjusted setting instructions.

[0189] Specific operation: When the user returns home, the living room lights will automatically change to a warm color and the settings will change to play their favorite music.

[0190] Step 4: Real-time environmental adjustment

[0191] The server analyzes external weather data and room usage patterns in real time and predictively adjusts the room temperature accordingly. The input is weather data and usage pattern data, and the output is room temperature adjustment instructions.

[0192] Specific operation: If the outside temperature is predicted to be high, the air conditioner will automatically turn on and adjust the set temperature appropriately.

[0193] Step 5: Monitor device usage and manage energy

[0194] The server monitors the usage of each device and optimizes energy consumption. The input is the usage data of each device, and the output is instructions to turn off unnecessary devices and functions.

[0195] Specific operation: The light will automatically turn off after a certain period of time has passed since the user last operated the light.

[0196] Step 6: Analyze and respond to health data

[0197] The device collects the user's sleep and activity data and sends it to a server. The input is the user's health data, and the output is the optimal environmental settings.

[0198] Specific operation: Before the user goes to sleep, the lighting changes to a warm color, automatically creating a comfortable sleeping environment.

[0199] Step 7: Execute Interactive Features

[0200] The server analyzes the camera data and recognizes the user's movements. The input is the camera image data, and the output is instructions for interactive game content.

[0201] Specific Action: Dynamic graphics are projected onto the wall as the user plays an exercise game with their family.

[0202] Step 8: Setting preferences based on video content

[0203] The server obtains the data of the video content being played on the device and determines the optimal environment settings. The input is the video content data, and the output is instructions for the environment settings.

[0204] What it does: When you play a yoga video, the lighting changes to a soft, relaxing color and the room temperature is adjusted appropriately.

[0205] Step 9: Optimize your remote work environment

[0206] The server analyzes the user's remote conference schedule and provides instructions for optimal environment settings. The input is the remote conference schedule data, and the output is instructions for environment settings suitable for the conference.

[0207] Specific operation: The projector automatically starts up 15 minutes before the start of the meeting, and the brightness and sound settings are adjusted appropriately for the meeting.

[0208] (Application example 1)

[0209] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0210] While conventional systems were able to make users' living spaces interactive, they still had challenges in personalizing the customer experience in physical stores and providing individual product information and navigation using augmented reality technology. Furthermore, it was difficult to automatically adjust the in-store environment based on the customer's profile, making it impossible to provide a comfortable shopping experience with appropriate environmental settings.

[0211] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0212] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, means for personalizing the in-store environment to suit the customer's preferences based on the store's customer profile, means for providing product information to customers using augmented reality technology, providing detailed information on related products and navigating the store, and means for setting the optimal environment based on the analysis of the customer's profile. This makes it possible to improve the customer experience in physical stores and provide the optimal shopping environment for each individual customer.

[0213] "User behavior patterns and preferences" refer to the past repeated behavior and personal preference settings of a particular user.

[0214] "Means for automatically adjusting lighting, temperature, music, and video" refers to a mechanism that automatically changes or sets related environmental elements (lighting, temperature, music, and video) based on the user's settings and behavioral patterns.

[0215] "Means for analyzing behavioral patterns and outside temperatures and predictively adjusting the room temperature to an appropriate level" refers to a function that analyzes user behavioral data and external weather information, predicts the ideal room temperature, and adjusts it in advance.

[0216] "Means for providing interactive entertainment in response to user responses and selections" refers to a function that provides entertainment that changes dynamically based on the user's real-time responses and selections.

[0217] "Means to personalize the in-store environment to suit customer preferences based on the store's customer profile" refers to a system that adjusts the in-store environment, including lighting, temperature, and music, based on the profile information of customers who visit the store.

[0218] "Means for providing customers with product information, detailed guidance on related products, and in-store navigation using augmented reality technology" refers to a method or device that uses augmented reality (AR) technology to display product information in the customer's field of vision, providing guidance on related products, and supporting in-store navigation.

[0219] "Means for optimal environmental settings based on customer profile analysis" refers to a function that analyzes customer profile data and automatically adjusts ideal environmental settings (lighting, temperature, music, etc.) based on that data.

[0220] System configuration

[0221] A system for implementing this invention includes the following main elements:

[0222] 1. Server: This is the central control device that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[0223] 2. Terminal: A device operated by the user (smartphone, tablet, smart glasses) that communicates with the server and sends and receives user profile and configuration information.

[0224] 3. Devices: Lighting, temperature control, music playback, augmented reality devices, and other devices that adjust the physical environment and enhance the user's experience.

[0225] System Operation

[0226] Preparing the interface

[0227] The server connects to each device at system startup to obtain status and configuration information. The device connects to the server via an application, and the user profile is loaded. The system is then ready to configure itself based on the user's individual information.

[0228] Providing a personalized experience

[0229] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. It analyzes the user profile and determines the appropriate settings based on the results, sending them to the terminal and each device.

[0230] Examples:

[0231] If a customer prefers dim lighting when watching a movie, the server can automatically dim the lighting when the movie is playing.

[0232] Predictive Comfort Management

[0233] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[0234] Examples:

[0235] If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[0236] Interactive Entertainment

[0237] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[0238] Examples:

[0239] During the climax of a horror movie, the lighting is automatically set to red and the sound effects are emphasized.

[0240] Personalized environment based on store customer profiles

[0241] The server analyzes the profiles of customers who visit the store and personalizes the in-store environment to suit their preferences, allowing customers to enjoy the optimal shopping environment.

[0242] Examples:

[0243] By playing music that customers like and adjusting lighting and temperature appropriately, we provide a comfortable shopping experience.

[0244] Augmented reality technology for product information and navigation

[0245] The device uses augmented reality technology to display product information to the customer's visual field, providing detailed information on related products and navigation.

[0246] Examples:

[0247] As customers walk through the store using the smart glasses, a guide displayed on the glasses navigates them to the products they want to purchase and displays product features and pricing information.

[0248] Prompt Sentence Examples

[0249] "Automatically adjust the in-store environment (lighting, music, temperature) based on the user's profile. Also, if a customer is wearing smart glasses, show them detailed information about related products and help them navigate to the products they are looking for."

[0250] In this way, the system of the present invention can interactively and automatically adjust environmental settings and entertainment based on user behavior and preferences to enhance the in-store customer experience.

[0251] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0252] Step 1:

[0253] The server connects to each device at system startup and obtains status and configuration information.

[0254] Input: Identification and initial configuration data for each device.

[0255] Output: Current status and configuration information for each device.

[0256] The server collects the setting information sent by each device and adjusts the initial settings based on this.

[0257] Step 2:

[0258] The device connects to the server via the application and the user profile is loaded.

[0259] Input: User credentials and profile data.

[0260] Output: The user's individual profile.

[0261] The terminal launches the application and connects to the server to request and retrieve the user's profile.

[0262] Step 3:

[0263] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's behavioral patterns and preferences.

[0264] Input: User behavioral pattern data and preference settings.

[0265] Output: Adjustment instructions for each device.

[0266] The server analyzes the user's past behavioral patterns and preferences, and based on this calculates appropriate lighting, temperature, music, and video settings and sends them to the relevant devices.

[0267] Step 4:

[0268] The server analyzes external weather data and room usage patterns in real time and predictively adjusts the room temperature to an appropriate level.

[0269] Input: External weather data and room usage pattern data.

[0270] Output: Room temperature control command.

[0271] The server acquires external weather data, compares it with the user's room usage pattern, predicts the room temperature adjustment value, and sends it to the temperature adjustment device.

[0272] Step 5:

[0273] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[0274] Input: entertainment content data and user reaction data.

[0275] Output: Instructions to change lighting and sound settings.

[0276] The device plays entertainment, and the server collects user reactions in real time from sensor data and adjusts lighting and sound accordingly.

[0277] Step 6:

[0278] The server analyzes the profiles of customers who visit the store and personalizes the in-store environment to suit their preferences.

[0279] Input: Customer profile data.

[0280] Output: Adjustment instructions to each device in the store.

[0281] The server analyzes the profile information of customers visiting the store and issues instructions to each device in the store to adjust lighting, music, temperature, etc. based on the results.

[0282] Step 7:

[0283] The device uses augmented reality technology to display product information to the customer's visual field, providing detailed information on related products and navigation.

[0284] Input: Customer navigation request and product information data.

[0285] Output: Augmented reality display data.

[0286] The terminal (such as smart glasses) retrieves product information from the server based on the customer's request, displays it using augmented reality technology, and guides the customer to the location of the desired product.

[0287] This will improve the customer experience in physical stores and provide an optimal shopping environment for each individual customer.

[0288] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0289] The system of the present invention enables more advanced personalization by combining means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for predictively adjusting the room temperature by analyzing the outside temperature, and means for providing interactive entertainment, as well as an emotion engine that recognizes the user's emotions.

[0290] System configuration

[0291] The system includes the following main elements:

[0292] 1. Server: The central control unit that analyzes user behavior patterns, preferences, and emotions, and manages and adjusts various settings.

[0293] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[0294] 3. Emotion engine: An analysis device that recognizes emotions from the user's facial expressions, voice, text data, etc. and reflects them in the settings of the entire system.

[0295] 4. Lighting, thermostats, smart speakers, etc.: Devices for adjusting the physical environment, including integrated devices such as the "Smart Ceiling Nexus."

[0296] System Operation

[0297] Preparing the interface

[0298] When the system starts up, the server connects to "SmartCeiling Nexus" to obtain device status and configuration information. The device (smartphone or tablet) connects to the server via the application, and the user profile is loaded. This prepares the system to configure settings based on the user's individual information.

[0299] Providing a personalized experience

[0300] The server determines personalized settings for lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes data from the user profile and emotion engine, determines appropriate settings based on the results, and sends them to the device and "SmartCeiling Nexus."

[0301] Example: If the emotion engine detects that the user is feeling stressed, the server will set soft lighting and calming music to enhance relaxation.

[0302] Predictive Comfort Management

[0303] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[0304] Example: If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[0305] Interactive Entertainment

[0306] As entertainment content is played on the terminal, the server analyzes the user's responses, preferences and emotions in real time and dynamically changes lighting and sound settings.

[0307] Example: During the climax of a horror movie, if the system detects that the user is feeling scared, it will automatically turn the lights red and enhance the sound effects.

[0308] Optimizing energy efficiency

[0309] The server monitors the usage of each device and optimizes energy consumption by issuing instructions to the SmartCeiling Nexus to turn off unnecessary devices and functions, minimizing energy consumption.

[0310] Example: Turning off the lights automatically after a certain time has passed after the user leaves the room.

[0311] Health and Wellness Support

[0312] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and configures it to provide an appropriate environment.

[0313] Example: Before bedtime, if the emotion engine detects that the user is in a state of relaxation, it will change the lighting to a warmer color and keep the temperature optimal.

[0314] Interactive wall / ceiling games

[0315] The server analyzes the camera data and works in conjunction with the emotion engine to recognize the user's movements and emotions. Based on this, interactive game content is sent in real time to the "SmartCeiling Nexus" and projected onto the walls and ceiling.

[0316] Example: When a user plays a jogging game, if the emotion engine recognizes that the user is having fun, it will randomly add fun elements (light and sound effects).

[0317] Environment-based visual experiences

[0318] The server obtains data on the video content being played on the device, determines the optimal environmental settings (lighting, temperature, music), and issues instructions to "SmartCeiling Nexus."

[0319] Example: When playing a yoga video, if the emotion engine detects a strong sense of relaxation, it will soften the lighting and set the temperature optimally.

[0320] Optimizing remote work

[0321] The server analyzes the user's remote meeting schedule and provides the optimal environment settings. "SmartCeiling Nexus" starts the projector and adjusts the lighting and sound for the remote meeting.

[0322] Example: Before a remote meeting begins, if the emotion engine detects stress, it will turn the lights up brighter and play calming music in the background.

[0323] These functions allow users to enjoy a comfortable and efficient living environment. The system of the present invention adjusts the environment interactively and automatically, providing an optimal environment according to the user's needs and emotions.

[0324] The processing flow will be explained below.

[0325] Step 1:

[0326] The server connects to the SmartCeiling Nexus when the system starts up and obtains device status and configuration information.

[0327] Step 2:

[0328] The terminal starts a dedicated application and transmits user authentication information to the server.

[0329] Step 3:

[0330] After verifying the authentication information, the server reads the user profile and analyzes past behavioral patterns and preference data.

[0331] Step 4:

[0332] The emotion engine analyzes the user's facial expressions and voice and generates emotion data.

[0333] Step 5:

[0334] The server determines personalized settings such as lighting, temperature, music, and video based on the analysis results and emotional data.

[0335] Step 6:

[0336] The setting information determined by the server is sent to the terminal and "SmartCeiling Nexus."

[0337] Step 7:

[0338] "SmartCeiling Nexus" configures the physical environment, such as adjusting the color temperature of lighting, based on the setting information it receives.

[0339] Step 8:

[0340] The server collects real-time external weather data and room usage patterns to predictively determine a comfortable room temperature.

[0341] Step 9:

[0342] Based on instructions from the server, "SmartCeiling Nexus" automatically adjusts the heating and cooling settings.

[0343] Step 10:

[0344] When entertainment content is played on the terminal, the playback information is sent to the server.

[0345] Step 11:

[0346] The emotion engine analyzes users' reactions in real time while watching entertainment and generates emotion data.

[0347] Step 12:

[0348] The server sends instructions to the SmartCeiling Nexus to dynamically change lighting and sound settings based on the type of content and emotional data.

[0349] Step 13:

[0350] The SmartCeiling Nexus adjusts lighting and sound to provide an interactive entertainment experience.

[0351] Step 14:

[0352] The server monitors the energy usage of each device and issues instructions to turn off unnecessary devices.

[0353] Step 15:

[0354] "SmartCeiling Nexus" follows instructions from the server and automatically changes settings to optimize energy efficiency (e.g., turning off lights that have not been used for a long time).

[0355] Step 16:

[0356] The user's sleep and activity data collected on the device is sent to a server.

[0357] Step 17:

[0358] The emotion engine analyzes the user's emotions before sleep and checks their state of relaxation.

[0359] Step 18:

[0360] The server analyzes this data and emotional data to determine settings (e.g., lighting, temperature) to provide an optimal sleeping environment.

[0361] Step 19:

[0362] The server sends instructions to the SmartCeiling Nexus, which adjusts the color temperature of the lighting and room temperature to create an optimal sleeping environment.

[0363] Step 20:

[0364] The server detects the start of an interactive game and analyzes real-time video data from the camera.

[0365] Step 21:

[0366] The emotion engine analyzes the user's emotions during the game in real time and generates emotion data.

[0367] Step 22:

[0368] The server transmits the game content to the "SmartCeiling Nexus" in real time, where it is projected onto the walls and ceiling.

[0369] Step 23:

[0370] Data of the video content to be played on the terminal is transmitted to the server.

[0371] Step 24:

[0372] The emotion engine analyzes the user's emotions while watching the video and confirms that they are relaxed.

[0373] Step 25:

[0374] The server determines the optimal environmental settings for the video and sends that setting information to "SmartCeiling Nexus."

[0375] Step 26:

[0376] SmartCeiling Nexus adjusts lighting, temperature and music based on user instructions to provide an environment for relaxation and improved concentration.

[0377] Step 27:

[0378] The server analyzes the user's calendar information and checks for remote meeting schedules.

[0379] Step 28:

[0380] The emotion engine analyzes the user's emotions before a remote meeting and confirms that they are feeling stressed.

[0381] Step 29:

[0382] The server determines the optimal environment settings for the remote meeting and sends instructions to the SmartCeiling Nexus.

[0383] Step 30:

[0384] SmartCeiling Nexus activates the projector and adjusts lighting and sound for remote meetings.

[0385] Each of these processing steps taken allows the system to provide an interactive and comfortable living space for the user.

[0386] Example 2

[0387] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] Conventional environmental control systems have difficulty in fine-tuning the environment based on the user's behavioral patterns, preferences, and emotions, and are therefore inadequate in optimizing the environment to suit the user's condition in real time. Furthermore, they do not take into account efficient energy use, and are unable to provide interactive entertainment that reflects the user's health and emotions.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for analyzing the behavioral patterns and the outside temperature and predictively adjusting the room temperature to an appropriate level, means for recognizing the user's emotions and providing personalized settings according to the emotions, means for providing interactive entertainment according to the user's reactions and selections, and means for monitoring energy consumption and optimizing energy efficiency by turning off unnecessary devices. This makes it possible to optimize the environment based on the user's real-time state, improve energy efficiency, and provide interactive entertainment according to the user's emotions and health state.

[0390] "User behavior patterns" refer to the behavioral tendencies and routines that a user takes in their daily life, such as the time they return home, the time they go to bed, and the devices they use.

[0391] "Preferences" refer to personal tastes and preferences that a user has for a particular environmental setting, such as a favorite genre of music or a preferred lighting brightness.

[0392] "Lighting" refers to an artificial light source used to brighten a room or space, and whose brightness, color, flashing pattern, etc. can be adjusted.

[0393] "Temperature" refers to the temperature of a room or space, and can be adjusted using air conditioning, heating, cooling, etc.

[0394] "Music" is a combination of sounds reproduced from a sound source, and is selected and reproduced based on the user's preferences.

[0395] "Video" refers to visual content that is presented to users through devices such as televisions, projectors, and monitors.

[0396] "Outside temperature" refers to the temperature of the air outside, and is obtained using weather forecasts and real-time weather data.

[0397] "Emotion" refers to the user's state of mind, such as feelings of joy, sadness, anger, surprise, etc.

[0398] "Emotion recognition" means using sensors and analysis engines to identify emotions from a user's facial expressions, tone of voice, text input, etc.

[0399] "Personalized settings" refers to environmental settings that match a user's specific preferences, behavioral patterns, and emotions, and includes lighting, music, temperature, and video settings that are optimized for each individual user.

[0400] "Real-time monitoring" refers to monitoring that occurs at the present moment, and involves the immediate acquisition and analysis of data from sensors and devices.

[0401] "Optimizing energy efficiency" means minimizing energy consumption and using resources in a sustainable manner.

[0402] "Interactive entertainment" refers to entertainment content that changes depending on the user's reactions and choices, allowing for interactive enjoyment.

[0403] The system of the present invention automatically adjusts environmental settings based on a user's behavioral patterns and preferences, and further provides personalized settings by recognizing the user's emotions in real time using an emotion engine. The system includes the following main components:

[0404] 1. Server:

[0405] The server is the central control device of the system, analyzing user behavior patterns and emotional data to optimize environmental settings. The server is connected to an integrated device called "SmartCeiling Nexus" and acquires device status and setting information. The server dynamically changes environmental settings such as lighting, temperature, music, and video using user profiles and real-time sensor data.

[0406] 2. Terminal:

[0407] The terminal (smartphone or tablet) is the device operated by the user and is responsible for communicating with the server. The user logs in to the system using the terminal and sends their profile and settings information to the server, which allows the server to understand the user's preferences and behavioral patterns.

[0408] 3. Emotion Engine:

[0409] The emotion engine is a device that analyzes the user's facial expressions, voice, text data, etc. to recognize emotions. The emotion engine uses sensors such as cameras and microphones to collect real-time data and analyzes the user's current emotional state based on that data. The analysis results are sent to a server and used to optimize environment settings.

[0410] 4. Environmental control device:

[0411] Environmental control devices such as lighting, air conditioning, and smart speakers are integrated into the "SmartCeiling Nexus," which adjusts the physical environment based on instructions from the server.

[0412] To illustrate, consider the following scenario:

[0413] Scenario 1: Adjusting your environment when you get home

[0414] When a user returns home from work, the device connects to the home Wi-Fi and sends a notification to the server. The server reads the user's profile and analyzes the data obtained from the emotion engine. For example, if the user is feeling stressed, the server instructs the SmartCeiling Nexus to set soft lighting and relaxing music.

[0415] Example prompt sentence:

[0416] "Users are coming home and feeling stressed, so set up soft lighting and calming music to promote relaxation."

[0417] Scenario 2: Entertainment optimization

[0418] When a user watches a horror movie, the device sends playback information to the server. The server uses data from the emotion engine to analyze the user's reaction in real time. For example, if the user is feeling scared, the server instructs SmartCeiling Nexus to dim the lights and enhance the sound effects.

[0419] Example prompt sentence:

[0420] "The user is feeling scared during the climax of a horror movie, so dim the lights and amplify the sound effects."

[0421] Scenario 3: Health Support

[0422] Before the user goes to sleep, activity data is sent from the smartwatch to the server via the device. The server then uses this data and the analysis results of the emotion engine to provide an optimal sleeping environment, for example by turning the lighting to a warm color and setting the room temperature to a comfortable temperature.

[0423] Example prompt sentence:

[0424] "Because users are experiencing relaxation before bed, change the lighting to warmer colors and set the room temperature optimally."

[0425] This system can provide an optimal environment according to the user's real-time condition and improve energy efficiency. In particular, the introduction of an emotion engine enables fine-tuned adjustments to match the user's emotions, which is expected to significantly improve quality of life.

[0426] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0427] Step 1: Initialization and Connection

[0428] When the system starts up, the server first connects to the "environmental adjustment device" (e.g., "SmartCeiling Nexus") to obtain device status and configuration information. This initialization procedure prepares the system.

[0429] Input: Device identification information, network settings

[0430] Data processing: Capture device status information and store it in a database on the server.

[0431] Output: Connection status log, current settings information

[0432] Specific operation: The server detects the "SmartCeiling Nexus" via the network, obtains the status of LED lighting, air conditioning, music playback devices, etc., and creates a log that stores the information.

[0433] Step 2: Loading the User Profile

[0434] The terminal transmits the user's login information to the server, and the server retrieves the user profile and reads the user's preferences and behavioral patterns.

[0435] Input: User authentication information, User ID

[0436] Data processing: Reading and analyzing user profile information from the database

[0437] Output: User's past behavior patterns and preference settings

[0438] Specific operation: When a user logs in with a smartphone, the device sends the user ID and authentication information to the server, which then retrieves relevant information from the database and activates the user profile.

[0439] Step 3: Real-time data collection and emotion recognition

[0440] The emotion engine collects real-time data from sensors such as cameras and microphones, analyzes the user's emotions, and transmits this information to a server.

[0441] Input: Real-time video, audio, and text data

[0442] Data processing: Input data into the emotion analysis model to classify the emotional state

[0443] Output: User's emotional state (e.g., stressed, relaxed, happy)

[0444] Specific operation: When a user enters a room, the emotion engine collects data from the camera and microphone, performs emotion analysis in real time, and sends the results to the server.

[0445] Step 4: Optimize and adjust settings

[0446] The server optimizes specific environmental settings based on the user's past behavioral patterns, real-time data, and emotional state, and transmits the settings information to an "environment adjustment device."

[0447] Input: User profile, real-time emotional data, behavioral patterns, outside temperature data

[0448] Data processing: Comprehensive analysis of input data and calculation of optimal environment settings

[0449] Output: Optimized lighting, temperature, music, and video settings instructions

[0450] Specific operation: If the user is analyzed as feeling stressed, the server instructs the "environmental adjustment device" to set warm lighting and relaxing music.

[0451] Step 5: Sync your entertainment content

[0452] The device plays entertainment content and sends the information to the server, which then adjusts the environment settings based on the emotion data and content information.

[0453] Input: Entertainment content information currently being played, emotional data

[0454] Data processing: Analyzing the type of content and the user's emotional state to determine appropriate environment settings

[0455] Output: Adjusted lighting and sound settings

[0456] Specific operation: When a user watches a horror movie, the device sends playback information to the server, and the server analyzes the emotion engine data and sends instructions to the "environment adjustment device" to dim the lights and emphasize the sound effects.

[0457] Step 6: Monitor and optimize energy efficiency

[0458] The server monitors the usage of each device and optimizes energy efficiency by automatically turning off unnecessary devices.

[0459] Input: Operating status and energy consumption data of each device

[0460] Data processing: Analyze usage patterns and energy consumption to identify unnecessary devices

[0461] Output: Instructions to turn off devices, energy usage reports

[0462] Specific operation: After the user leaves the room, the server sends an instruction to the "environmental control device" to turn off all unnecessary lights and devices after a certain period of time.

[0463] Step 7: Monitor your health

[0464] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and provides appropriate settings.

[0465] Input: User's sleep data, activity data

[0466] Data processing: Analyze health data and determine optimal environmental settings

[0467] Output: Adjustable lighting and temperature settings

[0468] Specific operation: The device sends activity data obtained by the user from the smartwatch to the server, and the server adjusts the lighting and temperature based on that data.

[0469] Step 8: Controlling the Interactive Game

[0470] The server analyzes the camera data and works in conjunction with an emotion engine to recognize the user's movements and emotions, and provides interactive game content based on this.

[0471] Input: Real-time camera data, emotion data

[0472] Data processing: Analyzing user movements and emotions to adjust the game's appearance and behavior

[0473] Output: Instructions for changing game content, setting visual and sound effects

[0474] How it works: When a user plays an immersive game, the camera and emotion engine collect and analyze data, and based on the results, send instructions to the "environment adjustment device" to adjust effects and difficulty in real time.

[0475] Step 9: Delivering an environment-based visual experience

[0476] The server acquires data on the video content being played on the terminal, determines the optimal environmental settings, and issues instructions to the "environment adjustment device."

[0477] Input: Video content information, real-time emotion data

[0478] Data processing: Analyze content attributes and user sentiment to calculate optimal environment settings

[0479] Output: Adjusted lighting, temperature, music settings

[0480] How it works: When a user plays a yoga video, the server recognizes it and sends instructions to the "environmental control device" to adjust the lighting and temperature to enhance the relaxation effect.

[0481] Step 10: Optimize your remote work environment

[0482] The server analyzes the user's remote meeting schedule and instructs the "environmental adjustment device" on the optimal environment settings based on that.

[0483] Input: Remote meeting schedule information, emotion data

[0484] Data processing: Analyze meeting time and user sentiment to determine appropriate environment settings

[0485] Output: Adjusted lighting, projection settings, and sound settings

[0486] Specific operation: When a user inputs a remote meeting schedule into the device, the server analyzes the information and sends instructions to the "environmental adjustment device" to brighten the lights and play soothing music in the background before the meeting begins.

[0487] These processing steps enable the system to optimize the environment based on the user's real-time condition, providing a comfortable and efficient living environment.

[0488] (Application example 2)

[0489] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0490] While conventional environmental control systems can adjust the environment based on a user's behavioral patterns and the external environment, they lack the ability to dynamically adjust the environment based on the user's emotional state. Furthermore, they lack the ability to link with content distribution services, limiting the personalization of the user experience. The present invention aims to address these issues by integrating content distribution with highly accurate environmental adjustments based on the user's emotions.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0492] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, and means for recognizing the user's emotions and automatically setting the optimal environment during content delivery, thereby enabling highly personalized environmental adjustment according to the user's emotional state and recommendation and delivery of content based on emotions.

[0493] "User behavior patterns" refers to records of the user's daily activities and behaviors and their tendencies.

[0494] "Preferences" refer to the preferences or tastes that a user has for a particular condition or option.

[0495] An "emotion engine" is a system for recognizing and analyzing emotions from a user's facial expressions, voice, and text data.

[0496] "Lighting" refers to electrical equipment and its functions for adjusting the intensity and color tone of light in a room or space.

[0497] "Temperature" is a parameter for measuring and controlling the hotness or coldness of air and objects within a particular space.

[0498] "Music" is an art form expressed through the combination of sounds, and includes elements such as rhythm, melody, and harmony.

[0499] "Video" means a moving or static visual representation that is displayed on a screen or display.

[0500] "Outside temperature" refers to the temperature of the environment outside the building.

[0501] "Interactive entertainment" refers to entertainment content that users can enjoy through two-way interaction with the system.

[0502] "Content distribution" refers to the provision of media content such as video and music to users over a digital network such as the Internet.

[0503] A "recommendation engine" is a system that suggests appropriate content based on a user's past behavior, preferences, and emotions.

[0504] To implement this invention, it is necessary to build a system that includes the following main elements:

[0505] System configuration

[0506] The system includes the following elements:

[0507] 1. Server: This is the central control unit that analyzes user behavior patterns, preferences, and emotions, and manages and adjusts various settings. The server analyzes data from user profiles and emotion engines, and issues instructions for appropriate environment settings and content delivery.

[0508] 2. Terminal: A device operated by the user (e.g., smartphone, tablet) that communicates with the server to send and receive user profile and setting information. The user's emotional state is also acquired from this device.

[0509] 3. Emotion Engine: An analysis device that recognizes emotions from the user's facial expressions, voice, text data, etc. and reflects them in the overall system settings. It uses OpenCV and dlib, Google Speech-to-Text API for voice analysis, and natural language processing (NLP) models for text analysis.

[0510] 4. Environmental control devices: These are devices that adjust the physical environment, such as lighting, thermostats, and smart speakers. They are controlled using the Philips Hue API or IFTTT.

[0511] System Operation

[0512] User Profile Settings

[0513] The server connects to the environmental control device when the system starts up and obtains the device's status and setting information. The terminal communicates with the server and reads the user profile, and the system prepares to set the environment based on the user's individual information.

[0514] Emotion-based environmental adjustment

[0515] The server analyzes the user's emotional data obtained from the emotion engine and, based on the results, issues instructions to adjust the environment, such as lighting, temperature, music, and video. For example, if the user is feeling stressed, the server will change the lighting to a softer color and play relaxing music.

[0516] Emotion-based content recommendation

[0517] The server uses a recommendation engine to recommend appropriate content (movies, music, images, etc.) based on the user's emotional state. For this purpose, it uses the Netflix Recommendation Engine and Spotify API.

[0518] Collaboration with environmental devices

[0519] The server manages the environmental control devices using the Philips Hue API and IFTTT, enabling dynamic environmental adjustments based on the user's emotions and behavioral patterns.

[0520] Specific examples

[0521] 1. Emotion recognition: The emotion engine analyzes the user's facial expression data and recognizes that the user is feeling "stressed."

[0522] 2. Content recommendation: The recommendation engine recommends relaxing music and movies to reduce stress.

[0523] 3. Environmental adjustment: The environmental control device changes the room lighting to a warmer color and sets the temperature to a comfortable level.

[0524] Example prompt sentence:

[0525] Describe a system that adjusts to a user's emotional state and environmental settings. If the user is stressed, the system selects comfortable lighting, relaxing music, and video streaming recommendations to soothe the user.

[0526] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0527] Step 1:

[0528] When the system starts up, the server connects to the environmental control device and obtains the device's status and setting information. The input information is the device's current status and settings, and the output information is the operating status and setting value of each device. Specifically, the server uses the Philips Hue API to obtain the current color tone and brightness of the lighting device, and checks the status of the thermostat via IFTTT.

[0529] Step 2:

[0530] The device communicates with the server and reads the user profile. The input information is the user's individual settings and past behavioral patterns, and the output information is the user profile data sent to the server. Specifically, the device collects emotional data and behavioral history from the user's smartphone and sends this to the server.

[0531] Step 3:

[0532] The server analyzes the user's emotional data obtained from the emotion engine. The input is the user's facial expression, voice, and text data, and the output is the emotion recognition results. Specifically, facial expression analysis is performed using "OpenCV" and "dlib," voice analysis is performed using "Google Speech-to-Text API," and the data is comprehensively analyzed using a "natural language processing (NLP) model" to determine the user's emotional state.

[0533] Step 4:

[0534] The server uses the emotion engine's analysis results to recommend appropriate content using a recommendation engine. The input is the emotion recognition results, and the output is a list of recommended content. Specifically, it uses the Netflix Recommendation Engine and Spotify API to recommend relaxing movies and music to reduce stress.

[0535] Step 5:

[0536] The server instructs the environmental control devices on optimal settings based on the analysis results of the emotion engine and the recommendations of the recommendation engine. The input is the emotional state and recommended content, and the output is specific setting instructions for each device. Specifically, it uses the Philips Hue API to change the color tone of the lighting to a warmer color and to set the thermostat to a comfortable temperature.

[0537] Step 6:

[0538] The device presents the settings and recommended content from the server to the user. The input is the settings and recommended content list from the server, and the output is data in a display format that the user can see and operate. Specifically, the settings and recommended movies and music are displayed in a pop-up on the smartphone's UI, allowing the user to select.

[0539] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0540] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0541] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0542] [Second embodiment]

[0543] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0544] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0545] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0546] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0547] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0548] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0549] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0550] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0551] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0552] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0553] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0554] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0555] The present invention relates to a system for making a user's living space more interactive and comfortable. Specific embodiments of the present invention will be described below.

[0556] System configuration

[0557] The system includes the following main elements:

[0558] 1. Server: A central control unit that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[0559] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[0560] 3. Lighting, thermostats, smart speakers, etc.: Devices for adjusting the physical environment, including integrated devices such as the "Smart Ceiling Nexus."

[0561] System Operation

[0562] Preparing the interface

[0563] When the system starts up, the server connects to "SmartCeiling Nexus" and obtains device status and configuration information. The device (smartphone or tablet) connects to the server via the application, and the user profile is loaded. This prepares the system to configure settings based on the user's individual information.

[0564] Providing a personalized experience

[0565] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes the user profile and determines appropriate settings based on the results, which are then sent to the device and "SmartCeiling Nexus."

[0566] Example: If a user prefers dim lighting when watching movies, the server can automatically dim the lighting when the movie is playing.

[0567] Predictive Comfort Management

[0568] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[0569] Example: If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[0570] Interactive Entertainment

[0571] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[0572] Example: Automatically set the lighting to red and the sound effects to emphasize the climax scene of a horror movie.

[0573] Optimizing energy efficiency

[0574] The server monitors the usage of each device and optimizes energy consumption by issuing instructions to the SmartCeiling Nexus to turn off unnecessary devices and functions, minimizing energy consumption.

[0575] Example: Turning off the lights automatically after a certain time has passed after the user leaves the room.

[0576] Health and Wellness Support

[0577] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and configures it to provide an appropriate environment.

[0578] Example: Automatically change the lighting to warmer colors and adjust the temperature before the user goes to sleep.

[0579] Interactive wall / ceiling games

[0580] The server analyzes the camera data, recognizes the user's movements, and sends interactive game content in real time to the "SmartCeiling Nexus," which projects it onto the walls and ceiling.

[0581] Example: When a user plays a jogging game, the course is projected onto a wall and the environment changes as the user moves.

[0582] Environment-based visual experiences

[0583] The server obtains data on the video content being played on the device, determines the optimal environmental settings (lighting, temperature, music), and issues instructions to "SmartCeiling Nexus."

[0584] Example: When playing a yoga video, adjust the lighting to a softer color and set the temperature appropriately to enhance the relaxation effect.

[0585] Optimizing remote work

[0586] The server analyzes the user's remote meeting schedule and provides the optimal environment settings. "SmartCeiling Nexus" starts the projector and adjusts the lighting and sound for the remote meeting.

[0587] Example: Before a remote meeting begins, start the projector and set the lighting appropriately to ensure a smooth start to the meeting.

[0588] These functions allow users to enjoy a comfortable and efficient living environment. The system of the present invention adjusts the environment interactively and automatically to provide an optimal environment according to the user's needs.

[0589] The processing flow will be explained below.

[0590] Step 1:

[0591] The server connects to the SmartCeiling Nexus when the system starts up and obtains device status and configuration information.

[0592] Step 2:

[0593] The terminal starts a dedicated application and transmits user authentication information to the server.

[0594] Step 3:

[0595] After verifying the authentication information, the server reads the user profile and analyzes data on past behavioral patterns and preferences.

[0596] Step 4:

[0597] The server determines personalized settings such as lighting, temperature, music, and video based on the analysis results.

[0598] Step 5:

[0599] The setting information determined by the server is sent to the terminal and "SmartCeiling Nexus."

[0600] Step 6:

[0601] "SmartCeiling Nexus" configures the physical environment, such as adjusting the color temperature of lighting, based on the setting information it receives.

[0602] Step 7:

[0603] The server collects real-time external weather data and room usage patterns to predictively determine a comfortable room temperature.

[0604] Step 8:

[0605] Based on instructions from the server, "SmartCeiling Nexus" automatically adjusts the heating and cooling settings.

[0606] Step 9:

[0607] When entertainment content is played on the terminal, the playback information is sent to the server.

[0608] Step 10:

[0609] The server sends instructions to the SmartCeiling Nexus to dynamically change lighting and sound settings based on the type of content and user preferences.

[0610] Step 11:

[0611] The SmartCeiling Nexus adjusts lighting and sound to provide an interactive entertainment experience.

[0612] Step 12:

[0613] The server monitors the energy usage of each device and issues instructions to turn off unnecessary devices.

[0614] Step 13:

[0615] "SmartCeiling Nexus" follows instructions from the server and automatically changes settings to optimize energy efficiency (e.g., turning off lights that have not been used for a long time).

[0616] Step 14:

[0617] The user's sleep and activity data collected on the device is sent to a server.

[0618] Step 15:

[0619] The server analyzes this data and determines settings (e.g., lighting, temperature) to provide an optimal sleeping environment.

[0620] Step 16:

[0621] The server sends instructions to the SmartCeiling Nexus, which adjusts the color temperature of the lighting and room temperature to create an optimal sleeping environment.

[0622] Step 17:

[0623] The server detects the start of an interactive game and analyzes real-time video data from the camera.

[0624] Step 18:

[0625] Game content is sent from the server to the "SmartCeiling Nexus" in real time and projected onto the walls and ceiling.

[0626] Step 19:

[0627] Data of the video content to be played on the terminal is transmitted to the server.

[0628] Step 20:

[0629] The server determines the optimal environmental settings for the video and sends that setting information to "SmartCeiling Nexus."

[0630] Step 21:

[0631] SmartCeiling Nexus adjusts lighting, temperature and music based on user instructions to provide an environment for relaxation and improved concentration.

[0632] Step 22:

[0633] The server analyzes the user's calendar information and checks for remote meeting schedules.

[0634] Step 23:

[0635] The server determines the optimal environment settings for the remote meeting and sends instructions to the SmartCeiling Nexus.

[0636] Step 24:

[0637] SmartCeiling Nexus activates the projector and adjusts lighting and sound for remote meetings.

[0638] Each of these processing steps taken allows the system to provide an interactive and comfortable living space for the user.

[0639] Example 1

[0640] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0641] Modern living environments require multiple devices and systems to enhance user comfort and efficiency, but these often operate independently, making integrated management difficult. Furthermore, meeting a wide range of needs, such as optimizing energy consumption, supporting health, and providing interactive entertainment, requires a high degree of automation and precise configuration based on user profiles. Conventional systems struggle to meet these requirements and are unable to provide a fully comfortable living environment for users.

[0642] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0643] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, means for acquiring device status and setting information at system startup, means for reading a user profile and setting the settings based on the individual information, and means for monitoring the user's device usage and optimizing energy efficiency to minimize energy consumption at all times. This makes it possible to manage and improve the user's comfort, health, energy efficiency, and entertainment experience in an integrated manner.

[0644] A "behavioral pattern" refers to the actions or choices a user makes repeatedly in a particular environment or situation.

[0645] "Preferences" refer to the preferences or tendencies of a user that are particularly favored in a particular setting or environment.

[0646] "Lighting" refers to artificial light sources in a user's living space.

[0647] "Temperature" refers to the indoor temperature in the user's living space.

[0648] "Music" refers to audio content that is played in the user's living space.

[0649] "Video" refers to video and image content that is visually displayed in the user's living space.

[0650] "Outside temperature" refers to the temperature of the air outside the living space.

[0651] "Predictive adjustment" means predicting future changes based on past data and current conditions, and making appropriate settings in advance.

[0652] "Reaction" refers to the behavior or response a user exhibits in response to a particular stimulus or situation.

[0653] "Selection" refers to a user selecting a specific item or setting from multiple options.

[0654] "Interactive entertainment" refers to entertainment content that changes dynamically in response to user input and actions.

[0655] "Boot up" refers to getting a system or device up and running and ready for use.

[0656] "Device" refers to electronic devices and machines used in the user's living space.

[0657] "Configuration information" refers to the operating conditions and parameters of a device or system.

[0658] A "user profile" refers to a data set that includes a user's personal information, preferences, behavioral patterns, etc.

[0659] "Individual information" refers to information or data that is unique to a particular user.

[0660] "Monitoring" refers to the continuous watching and recording of the operating status of a particular device or system.

[0661] "Energy efficiency" refers to the efficiency with which a particular device or system uses the energy it consumes.

[0662] The present invention relates to a system that makes a user's living space more interactive and comfortable. This system is mainly composed of a server, a terminal, and various devices (lighting, temperature control devices, smart speakers, etc.). An embodiment of the present invention will now be described in detail.

[0663] System configuration

[0664] The system includes the following main elements:

[0665] 1. Server: A central control unit that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[0666] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[0667] 3. Various devices: Devices for adjusting the physical environment, including "integrated devices" and "sensor clusters."

[0668] System Operation

[0669] Preparing the interface

[0670] When the system starts up, the server connects to the "integrated devices" and obtains the status and setting information of each device. This determines the initial state of the entire system. The device (smartphone or tablet) connects to the server via an application, and the user profile is loaded. This prepares the system to perform settings based on the user's individual information.

[0671] Example: When the system starts up, it receives information about the current room temperature and lighting settings from the integrated device. When you start the app on your smartphone and log in, the server retrieves the user's past activity log and settings.

[0672] Providing a personalized experience

[0673] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes the user profile and determines appropriate settings based on the results, which are then sent to the terminal and each device.

[0674] Example: When a user returns home, the living room lights automatically change to a warmer color and the settings are changed to play their favorite music.

[0675] Predictive Comfort Management

[0676] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[0677] Example: If the outside temperature is predicted to be high, the air conditioner will automatically turn on and adjust the set temperature appropriately.

[0678] Device usage monitoring and energy management

[0679] The server monitors the usage of each device and optimizes energy consumption, issuing instructions to turn off unnecessary devices and functions.

[0680] Example: Lights turn off automatically after a certain time has passed since the last time the user operated them.

[0681] Health data analysis and response

[0682] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and sets optimal settings.

[0683] Example: When the user goes to sleep, the lights change to warm colors to automatically create a comfortable sleeping environment.

[0684] Executing interactive functions

[0685] The server analyzes the camera data, recognizes the user's movements, and sends interactive game content in real time to the "integrated device," which projects it onto the walls and ceiling.

[0686] Example: Dynamic graphics are projected onto the wall as a user plays an exercise game with their family.

[0687] Preferences based on video content

[0688] The server obtains data on the video content being played on the terminal, determines the optimal environment settings, and issues instructions to each device.

[0689] Example: When you play a yoga video, the lighting changes to a softer, more relaxing color and the room temperature adjusts appropriately.

[0690] Optimizing the remote work environment

[0691] The server analyzes the user's remote conference schedule and provides optimal environment settings, while the integrated device activates the projector and adjusts the lighting and sound for the remote conference.

[0692] Example: The projector automatically starts 15 minutes before the start of a meeting, and the brightness and sound settings are adjusted to suit the meeting.

[0693] With these functions, users can enjoy a comfortable and efficient living environment. The system adjusts the environment interactively and automatically to provide an optimal environment according to the user's needs.

[0694] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0695] Step 1: Boot up the system and connect devices

[0696] When the system starts up, the server connects to the integrated devices and obtains the status and setting information of each device. This determines the initial state of the entire system. The input is the system startup signal and the response from the integrated devices, and the output is the current status and setting information of each device.

[0697] How it works: When the system starts up, the server sends a signal to the integrated device and receives information about the current room temperature and lighting settings.

[0698] Step 2: Loading the User Profile

[0699] The device (smartphone or tablet) connects to the server via an application and reads the user profile. The input is the device's login information, and the output is profile information that includes the user's past behavioral data and preferences.

[0700] Specific operation: When a user launches the app on their smartphone and logs in, the server retrieves the user's past activity logs and settings.

[0701] Step 3: Apply personalized settings

[0702] The server analyzes the user profile and automatically adjusts lighting, temperature, music, video, etc. based on past behavioral patterns and preferences. The input is the user profile and current environmental data, and the output is automatically adjusted setting instructions.

[0703] Specific operation: When the user returns home, the living room lights will automatically change to a warm color and the settings will change to play their favorite music.

[0704] Step 4: Real-time environmental adjustment

[0705] The server analyzes external weather data and room usage patterns in real time and predictively adjusts the room temperature accordingly. The input is weather data and usage pattern data, and the output is room temperature adjustment instructions.

[0706] Specific operation: If the outside temperature is predicted to be high, the air conditioner will automatically turn on and adjust the set temperature appropriately.

[0707] Step 5: Monitor device usage and manage energy

[0708] The server monitors the usage of each device and optimizes energy consumption. The input is the usage data of each device, and the output is instructions to turn off unnecessary devices and functions.

[0709] Specific operation: The light will automatically turn off after a certain period of time has passed since the user last operated the light.

[0710] Step 6: Analyze and respond to health data

[0711] The device collects the user's sleep and activity data and sends it to a server. The input is the user's health data, and the output is the optimal environmental settings.

[0712] Specific operation: Before the user goes to sleep, the lighting changes to a warm color, automatically creating a comfortable sleeping environment.

[0713] Step 7: Execute Interactive Features

[0714] The server analyzes the camera data and recognizes the user's movements. The input is the camera image data, and the output is instructions for interactive game content.

[0715] Specific Action: Dynamic graphics are projected onto the wall as the user plays an exercise game with their family.

[0716] Step 8: Setting preferences based on video content

[0717] The server obtains the data of the video content being played on the device and determines the optimal environment settings. The input is the video content data, and the output is instructions for the environment settings.

[0718] What it does: When you play a yoga video, the lighting changes to a soft, relaxing color and the room temperature is adjusted appropriately.

[0719] Step 9: Optimize your remote work environment

[0720] The server analyzes the user's remote conference schedule and provides instructions for optimal environment settings. The input is the remote conference schedule data, and the output is instructions for environment settings suitable for the conference.

[0721] Specific operation: The projector automatically starts up 15 minutes before the start of the meeting, and the brightness and sound settings are adjusted appropriately for the meeting.

[0722] (Application example 1)

[0723] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] While conventional systems were able to make users' living spaces interactive, they still had challenges in personalizing the customer experience in physical stores and providing individual product information and navigation using augmented reality technology. Furthermore, it was difficult to automatically adjust the in-store environment based on the customer's profile, making it impossible to provide a comfortable shopping experience with appropriate environmental settings.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0726] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, means for personalizing the in-store environment to suit the customer's preferences based on the store's customer profile, means for providing product information to customers using augmented reality technology, providing detailed information on related products and navigating the store, and means for setting the optimal environment based on the analysis of the customer's profile. This makes it possible to improve the customer experience in physical stores and provide the optimal shopping environment for each individual customer.

[0727] "User behavior patterns and preferences" refer to the past repeated behavior and personal preference settings of a particular user.

[0728] "Means for automatically adjusting lighting, temperature, music, and video" refers to a mechanism that automatically changes or sets related environmental elements (lighting, temperature, music, and video) based on the user's settings and behavioral patterns.

[0729] "Means for analyzing behavioral patterns and outside temperatures and predictively adjusting the room temperature to an appropriate level" refers to a function that analyzes user behavioral data and external weather information, predicts the ideal room temperature, and adjusts it in advance.

[0730] "Means for providing interactive entertainment in response to user responses and selections" refers to a function that provides entertainment that changes dynamically based on the user's real-time responses and selections.

[0731] "Means to personalize the in-store environment to suit customer preferences based on the store's customer profile" refers to a system that adjusts the in-store environment, including lighting, temperature, and music, based on the profile information of customers who visit the store.

[0732] "Means for providing customers with product information, detailed guidance on related products, and in-store navigation using augmented reality technology" refers to a method or device that uses augmented reality (AR) technology to display product information in the customer's field of vision, providing guidance on related products, and supporting in-store navigation.

[0733] "Means for optimal environmental settings based on customer profile analysis" refers to a function that analyzes customer profile data and automatically adjusts ideal environmental settings (lighting, temperature, music, etc.) based on that data.

[0734] System configuration

[0735] A system for implementing this invention includes the following main elements:

[0736] 1. Server: This is the central control device that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[0737] 2. Terminal: A device operated by the user (smartphone, tablet, smart glasses) that communicates with the server and sends and receives user profile and configuration information.

[0738] 3. Devices: Lighting, temperature control, music playback, augmented reality devices, and other devices that adjust the physical environment and enhance the user's experience.

[0739] System Operation

[0740] Preparing the interface

[0741] The server connects to each device at system startup to obtain status and configuration information. The device connects to the server via an application, and the user profile is loaded. The system is then ready to configure itself based on the user's individual information.

[0742] Providing a personalized experience

[0743] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. It analyzes the user profile and determines the appropriate settings based on the results, sending them to the terminal and each device.

[0744] Examples:

[0745] If a customer prefers dim lighting when watching a movie, the server can automatically dim the lighting when the movie is playing.

[0746] Predictive Comfort Management

[0747] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[0748] Examples:

[0749] If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[0750] Interactive Entertainment

[0751] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[0752] Examples:

[0753] During the climax of a horror movie, the lighting is automatically set to red and the sound effects are emphasized.

[0754] Personalized environment based on store customer profiles

[0755] The server analyzes the profiles of customers who visit the store and personalizes the in-store environment to suit their preferences, allowing customers to enjoy the optimal shopping environment.

[0756] Examples:

[0757] By playing music that customers like and adjusting lighting and temperature appropriately, we provide a comfortable shopping experience.

[0758] Augmented reality technology for product information and navigation

[0759] The device uses augmented reality technology to display product information to the customer's visual field, providing detailed information on related products and navigation.

[0760] Examples:

[0761] As customers walk through the store using the smart glasses, a guide displayed on the glasses navigates them to the products they want to purchase and displays product features and pricing information.

[0762] Prompt Sentence Examples

[0763] "Automatically adjust the in-store environment (lighting, music, temperature) based on the user's profile. Also, if a customer is wearing smart glasses, show them detailed information about related products and help them navigate to the products they are looking for."

[0764] In this way, the system of the present invention can interactively and automatically adjust environmental settings and entertainment based on user behavior and preferences to enhance the in-store customer experience.

[0765] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0766] Step 1:

[0767] The server connects to each device at system startup and obtains status and configuration information.

[0768] Input: Identification and initial configuration data for each device.

[0769] Output: Current status and configuration information for each device.

[0770] The server collects the setting information sent by each device and adjusts the initial settings based on this.

[0771] Step 2:

[0772] The device connects to the server via the application and the user profile is loaded.

[0773] Input: User credentials and profile data.

[0774] Output: The user's individual profile.

[0775] The terminal launches the application and connects to the server to request and retrieve the user's profile.

[0776] Step 3:

[0777] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's behavioral patterns and preferences.

[0778] Input: User behavioral pattern data and preference settings.

[0779] Output: Adjustment instructions for each device.

[0780] The server analyzes the user's past behavioral patterns and preferences, and based on this calculates appropriate lighting, temperature, music, and video settings and sends them to the relevant devices.

[0781] Step 4:

[0782] The server analyzes external weather data and room usage patterns in real time and predictively adjusts the room temperature to an appropriate level.

[0783] Input: External weather data and room usage pattern data.

[0784] Output: Room temperature control command.

[0785] The server acquires external weather data, compares it with the user's room usage pattern, predicts the room temperature adjustment value, and sends it to the temperature adjustment device.

[0786] Step 5:

[0787] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[0788] Input: entertainment content data and user reaction data.

[0789] Output: Instructions to change lighting and sound settings.

[0790] The device plays entertainment, and the server collects user reactions in real time from sensor data and adjusts lighting and sound accordingly.

[0791] Step 6:

[0792] The server analyzes the profiles of customers who visit the store and personalizes the in-store environment to suit their preferences.

[0793] Input: Customer profile data.

[0794] Output: Adjustment instructions to each device in the store.

[0795] The server analyzes the profile information of customers visiting the store and issues instructions to each device in the store to adjust lighting, music, temperature, etc. based on the results.

[0796] Step 7:

[0797] The device uses augmented reality technology to display product information to the customer's visual field, providing detailed information on related products and navigation.

[0798] Input: Customer navigation request and product information data.

[0799] Output: Augmented reality display data.

[0800] The terminal (such as smart glasses) retrieves product information from the server based on the customer's request, displays it using augmented reality technology, and guides the customer to the location of the desired product.

[0801] This will improve the customer experience in physical stores and provide an optimal shopping environment for each individual customer.

[0802] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0803] The system of the present invention enables more advanced personalization by combining means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for predictively adjusting the room temperature by analyzing the outside temperature, and means for providing interactive entertainment, as well as an emotion engine that recognizes the user's emotions.

[0804] System configuration

[0805] The system includes the following main elements:

[0806] 1. Server: The central control unit that analyzes user behavior patterns, preferences, and emotions, and manages and adjusts various settings.

[0807] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[0808] 3. Emotion engine: An analysis device that recognizes emotions from the user's facial expressions, voice, text data, etc. and reflects them in the settings of the entire system.

[0809] 4. Lighting, thermostats, smart speakers, etc.: Devices for adjusting the physical environment, including integrated devices such as the "Smart Ceiling Nexus."

[0810] System Operation

[0811] Preparing the interface

[0812] When the system starts up, the server connects to "SmartCeiling Nexus" to obtain device status and configuration information. The device (smartphone or tablet) connects to the server via the application, and the user profile is loaded. This prepares the system to configure settings based on the user's individual information.

[0813] Providing a personalized experience

[0814] The server determines personalized settings for lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes data from the user profile and emotion engine, determines appropriate settings based on the results, and sends them to the device and "SmartCeiling Nexus."

[0815] Example: If the emotion engine detects that the user is feeling stressed, the server will set soft lighting and calming music to enhance relaxation.

[0816] Predictive Comfort Management

[0817] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[0818] Example: If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[0819] Interactive Entertainment

[0820] As entertainment content is played on the terminal, the server analyzes the user's responses, preferences and emotions in real time and dynamically changes lighting and sound settings.

[0821] Example: During the climax of a horror movie, if the system detects that the user is feeling scared, it will automatically turn the lights red and enhance the sound effects.

[0822] Optimizing energy efficiency

[0823] The server monitors the usage of each device and optimizes energy consumption by issuing instructions to the SmartCeiling Nexus to turn off unnecessary devices and functions, minimizing energy consumption.

[0824] Example: Turning off the lights automatically after a certain time has passed after the user leaves the room.

[0825] Health and Wellness Support

[0826] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and configures it to provide an appropriate environment.

[0827] Example: Before bedtime, if the emotion engine detects that the user is in a state of relaxation, it will change the lighting to a warmer color and keep the temperature optimal.

[0828] Interactive wall / ceiling games

[0829] The server analyzes the camera data and works in conjunction with the emotion engine to recognize the user's movements and emotions. Based on this, interactive game content is sent in real time to the "SmartCeiling Nexus" and projected onto the walls and ceiling.

[0830] Example: When a user plays a jogging game, if the emotion engine recognizes that the user is having fun, it will randomly add fun elements (light and sound effects).

[0831] Environment-based visual experiences

[0832] The server obtains data on the video content being played on the device, determines the optimal environmental settings (lighting, temperature, music), and issues instructions to "SmartCeiling Nexus."

[0833] Example: When playing a yoga video, if the emotion engine detects a strong sense of relaxation, it will soften the lighting and set the temperature optimally.

[0834] Optimizing remote work

[0835] The server analyzes the user's remote meeting schedule and provides the optimal environment settings. "SmartCeiling Nexus" starts the projector and adjusts the lighting and sound for the remote meeting.

[0836] Example: Before a remote meeting begins, if the emotion engine detects stress, it will turn the lights up brighter and play calming music in the background.

[0837] These functions allow users to enjoy a comfortable and efficient living environment. The system of the present invention adjusts the environment interactively and automatically, providing an optimal environment according to the user's needs and emotions.

[0838] The processing flow will be explained below.

[0839] Step 1:

[0840] The server connects to the SmartCeiling Nexus when the system starts up and obtains device status and configuration information.

[0841] Step 2:

[0842] The terminal starts a dedicated application and transmits user authentication information to the server.

[0843] Step 3:

[0844] After verifying the authentication information, the server reads the user profile and analyzes past behavioral patterns and preference data.

[0845] Step 4:

[0846] The emotion engine analyzes the user's facial expressions and voice and generates emotion data.

[0847] Step 5:

[0848] The server determines personalized settings such as lighting, temperature, music, and video based on the analysis results and emotional data.

[0849] Step 6:

[0850] The setting information determined by the server is sent to the terminal and "SmartCeiling Nexus."

[0851] Step 7:

[0852] "SmartCeiling Nexus" configures the physical environment, such as adjusting the color temperature of lighting, based on the setting information it receives.

[0853] Step 8:

[0854] The server collects real-time external weather data and room usage patterns to predictively determine a comfortable room temperature.

[0855] Step 9:

[0856] Based on instructions from the server, "SmartCeiling Nexus" automatically adjusts the heating and cooling settings.

[0857] Step 10:

[0858] When entertainment content is played on the terminal, the playback information is sent to the server.

[0859] Step 11:

[0860] The emotion engine analyzes users' reactions in real time while watching entertainment and generates emotion data.

[0861] Step 12:

[0862] The server sends instructions to the SmartCeiling Nexus to dynamically change lighting and sound settings based on the type of content and emotional data.

[0863] Step 13:

[0864] The SmartCeiling Nexus adjusts lighting and sound to provide an interactive entertainment experience.

[0865] Step 14:

[0866] The server monitors the energy usage of each device and issues instructions to turn off unnecessary devices.

[0867] Step 15:

[0868] "SmartCeiling Nexus" follows instructions from the server and automatically changes settings to optimize energy efficiency (e.g., turning off lights that have not been used for a long time).

[0869] Step 16:

[0870] The user's sleep and activity data collected on the device is sent to a server.

[0871] Step 17:

[0872] The emotion engine analyzes the user's emotions before sleep and checks their state of relaxation.

[0873] Step 18:

[0874] The server analyzes this data and emotional data to determine settings (e.g., lighting, temperature) to provide an optimal sleeping environment.

[0875] Step 19:

[0876] The server sends instructions to the SmartCeiling Nexus, which adjusts the color temperature of the lighting and room temperature to create an optimal sleeping environment.

[0877] Step 20:

[0878] The server detects the start of an interactive game and analyzes real-time video data from the camera.

[0879] Step 21:

[0880] The emotion engine analyzes the user's emotions during the game in real time and generates emotion data.

[0881] Step 22:

[0882] The server transmits the game content to the "SmartCeiling Nexus" in real time, where it is projected onto the walls and ceiling.

[0883] Step 23:

[0884] Data of the video content to be played on the terminal is transmitted to the server.

[0885] Step 24:

[0886] The emotion engine analyzes the user's emotions while watching the video and confirms that they are relaxed.

[0887] Step 25:

[0888] The server determines the optimal environmental settings for the video and sends that setting information to "SmartCeiling Nexus."

[0889] Step 26:

[0890] SmartCeiling Nexus adjusts lighting, temperature and music based on user instructions to provide an environment for relaxation and improved concentration.

[0891] Step 27:

[0892] The server analyzes the user's calendar information and checks for remote meeting schedules.

[0893] Step 28:

[0894] The emotion engine analyzes the user's emotions before a remote meeting and confirms that they are feeling stressed.

[0895] Step 29:

[0896] The server determines the optimal environment settings for the remote meeting and sends instructions to the SmartCeiling Nexus.

[0897] Step 30:

[0898] SmartCeiling Nexus activates the projector and adjusts lighting and sound for remote meetings.

[0899] Each of these processing steps taken allows the system to provide an interactive and comfortable living space for the user.

[0900] Example 2

[0901] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0902] Conventional environmental control systems have difficulty in fine-tuning the environment based on the user's behavioral patterns, preferences, and emotions, and are therefore inadequate in optimizing the environment to suit the user's condition in real time. Furthermore, they do not take into account efficient energy use, and are unable to provide interactive entertainment that reflects the user's health and emotions.

[0903] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for analyzing the behavioral patterns and the outside temperature and predictively adjusting the room temperature to an appropriate level, means for recognizing the user's emotions and providing personalized settings according to the emotions, means for providing interactive entertainment according to the user's reactions and selections, and means for monitoring energy consumption and optimizing energy efficiency by turning off unnecessary devices. This makes it possible to optimize the environment based on the user's real-time state, improve energy efficiency, and provide interactive entertainment according to the user's emotions and health state.

[0904] "User behavior patterns" refer to the behavioral tendencies and routines that a user takes in their daily life, such as the time they return home, the time they go to bed, and the devices they use.

[0905] "Preferences" refer to personal tastes and preferences that a user has for a particular environmental setting, such as a favorite genre of music or a preferred lighting brightness.

[0906] "Lighting" refers to an artificial light source used to brighten a room or space, and whose brightness, color, flashing pattern, etc. can be adjusted.

[0907] "Temperature" refers to the temperature of a room or space, and can be adjusted using air conditioning, heating, cooling, etc.

[0908] "Music" is a combination of sounds reproduced from a sound source, and is selected and reproduced based on the user's preferences.

[0909] "Video" refers to visual content that is presented to users through devices such as televisions, projectors, and monitors.

[0910] "Outside temperature" refers to the temperature of the air outside, and is obtained using weather forecasts and real-time weather data.

[0911] "Emotion" refers to the user's state of mind, such as feelings of joy, sadness, anger, surprise, etc.

[0912] "Emotion recognition" means using sensors and analysis engines to identify emotions from a user's facial expressions, tone of voice, text input, etc.

[0913] "Personalized settings" refers to environmental settings that match a user's specific preferences, behavioral patterns, and emotions, and includes lighting, music, temperature, and video settings that are optimized for each individual user.

[0914] "Real-time monitoring" refers to monitoring that occurs at the present moment, and involves the immediate acquisition and analysis of data from sensors and devices.

[0915] "Optimizing energy efficiency" means minimizing energy consumption and using resources in a sustainable manner.

[0916] "Interactive entertainment" refers to entertainment content that changes depending on the user's reactions and choices, allowing for interactive enjoyment.

[0917] The system of the present invention automatically adjusts environmental settings based on a user's behavioral patterns and preferences, and further provides personalized settings by recognizing the user's emotions in real time using an emotion engine. The system includes the following main components:

[0918] 1. Server:

[0919] The server is the central control device of the system, analyzing user behavior patterns and emotional data to optimize environmental settings. The server is connected to an integrated device called "SmartCeiling Nexus" and acquires device status and setting information. The server dynamically changes environmental settings such as lighting, temperature, music, and video using user profiles and real-time sensor data.

[0920] 2. Terminal:

[0921] The terminal (smartphone or tablet) is the device operated by the user and is responsible for communicating with the server. The user logs in to the system using the terminal and sends their profile and settings information to the server, which allows the server to understand the user's preferences and behavioral patterns.

[0922] 3. Emotion Engine:

[0923] The emotion engine is a device that analyzes the user's facial expressions, voice, text data, etc. to recognize emotions. The emotion engine uses sensors such as cameras and microphones to collect real-time data and analyzes the user's current emotional state based on that data. The analysis results are sent to a server and used to optimize environment settings.

[0924] 4. Environmental control device:

[0925] Environmental control devices such as lighting, air conditioning, and smart speakers are integrated into the "SmartCeiling Nexus," which adjusts the physical environment based on instructions from the server.

[0926] To illustrate, consider the following scenario:

[0927] Scenario 1: Adjusting your environment when you get home

[0928] When a user returns home from work, the device connects to the home Wi-Fi and sends a notification to the server. The server reads the user's profile and analyzes the data obtained from the emotion engine. For example, if the user is feeling stressed, the server instructs the SmartCeiling Nexus to set soft lighting and relaxing music.

[0929] Example prompt sentence:

[0930] "Users are coming home and feeling stressed, so set up soft lighting and calming music to promote relaxation."

[0931] Scenario 2: Entertainment optimization

[0932] When a user watches a horror movie, the device sends playback information to the server. The server uses data from the emotion engine to analyze the user's reaction in real time. For example, if the user is feeling scared, the server instructs SmartCeiling Nexus to dim the lights and enhance the sound effects.

[0933] Example prompt sentence:

[0934] "The user is feeling scared during the climax of a horror movie, so dim the lights and amplify the sound effects."

[0935] Scenario 3: Health Support

[0936] Before the user goes to sleep, activity data is sent from the smartwatch to the server via the device. The server then uses this data and the analysis results of the emotion engine to provide an optimal sleeping environment, for example by turning the lighting to a warm color and setting the room temperature to a comfortable temperature.

[0937] Example prompt sentence:

[0938] "Because users are experiencing relaxation before bed, change the lighting to warmer colors and set the room temperature optimally."

[0939] This system can provide an optimal environment according to the user's real-time condition and improve energy efficiency. In particular, the introduction of an emotion engine enables fine-tuned adjustments to match the user's emotions, which is expected to significantly improve quality of life.

[0940] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0941] Step 1: Initialization and Connection

[0942] When the system starts up, the server first connects to the "environmental adjustment device" (e.g., "SmartCeiling Nexus") to obtain device status and configuration information. This initialization procedure prepares the system.

[0943] Input: Device identification information, network settings

[0944] Data processing: Capture device status information and store it in a database on the server.

[0945] Output: Connection status log, current settings information

[0946] Specific operation: The server detects the "SmartCeiling Nexus" via the network, obtains the status of LED lighting, air conditioning, music playback devices, etc., and creates a log that stores the information.

[0947] Step 2: Loading the User Profile

[0948] The terminal transmits the user's login information to the server, and the server retrieves the user profile and reads the user's preferences and behavioral patterns.

[0949] Input: User authentication information, User ID

[0950] Data processing: Reading and analyzing user profile information from the database

[0951] Output: User's past behavior patterns and preference settings

[0952] Specific operation: When a user logs in with a smartphone, the device sends the user ID and authentication information to the server, which then retrieves relevant information from the database and activates the user profile.

[0953] Step 3: Real-time data collection and emotion recognition

[0954] The emotion engine collects real-time data from sensors such as cameras and microphones, analyzes the user's emotions, and transmits this information to a server.

[0955] Input: Real-time video, audio, and text data

[0956] Data processing: Input data into the emotion analysis model to classify the emotional state

[0957] Output: User's emotional state (e.g., stressed, relaxed, happy)

[0958] Specific operation: When a user enters a room, the emotion engine collects data from the camera and microphone, performs emotion analysis in real time, and sends the results to the server.

[0959] Step 4: Optimize and adjust settings

[0960] The server optimizes specific environmental settings based on the user's past behavioral patterns, real-time data, and emotional state, and transmits the settings information to an "environment adjustment device."

[0961] Input: User profile, real-time emotional data, behavioral patterns, outside temperature data

[0962] Data processing: Comprehensive analysis of input data and calculation of optimal environment settings

[0963] Output: Optimized lighting, temperature, music, and video settings instructions

[0964] Specific operation: If the user is analyzed as feeling stressed, the server instructs the "environmental adjustment device" to set warm lighting and relaxing music.

[0965] Step 5: Sync your entertainment content

[0966] The device plays entertainment content and sends the information to the server, which then adjusts the environment settings based on the emotion data and content information.

[0967] Input: Entertainment content information currently being played, emotional data

[0968] Data processing: Analyzing the type of content and the user's emotional state to determine appropriate environment settings

[0969] Output: Adjusted lighting and sound settings

[0970] Specific operation: When a user watches a horror movie, the device sends playback information to the server, and the server analyzes the emotion engine data and sends instructions to the "environment adjustment device" to dim the lights and emphasize the sound effects.

[0971] Step 6: Monitor and optimize energy efficiency

[0972] The server monitors the usage of each device and optimizes energy efficiency by automatically turning off unnecessary devices.

[0973] Input: Operating status and energy consumption data of each device

[0974] Data processing: Analyze usage patterns and energy consumption to identify unnecessary devices

[0975] Output: Instructions to turn off devices, energy usage reports

[0976] Specific operation: After the user leaves the room, the server sends an instruction to the "environmental control device" to turn off all unnecessary lights and devices after a certain period of time.

[0977] Step 7: Monitor your health

[0978] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and provides appropriate settings.

[0979] Input: User's sleep data, activity data

[0980] Data processing: Analyze health data and determine optimal environmental settings

[0981] Output: Adjustable lighting and temperature settings

[0982] Specific operation: The device sends activity data obtained by the user from the smartwatch to the server, and the server adjusts the lighting and temperature based on that data.

[0983] Step 8: Controlling the Interactive Game

[0984] The server analyzes the camera data and works in conjunction with an emotion engine to recognize the user's movements and emotions, and provides interactive game content based on this.

[0985] Input: Real-time camera data, emotion data

[0986] Data processing: Analyzing user movements and emotions to adjust the game's appearance and behavior

[0987] Output: Instructions for changing game content, setting visual and sound effects

[0988] How it works: When a user plays an immersive game, the camera and emotion engine collect and analyze data, and based on the results, send instructions to the "environment adjustment device" to adjust effects and difficulty in real time.

[0989] Step 9: Delivering an environment-based visual experience

[0990] The server acquires data on the video content being played on the terminal, determines the optimal environmental settings, and issues instructions to the "environment adjustment device."

[0991] Input: Video content information, real-time emotion data

[0992] Data processing: Analyze content attributes and user sentiment to calculate optimal environment settings

[0993] Output: Adjusted lighting, temperature, music settings

[0994] How it works: When a user plays a yoga video, the server recognizes it and sends instructions to the "environmental control device" to adjust the lighting and temperature to enhance the relaxation effect.

[0995] Step 10: Optimize your remote work environment

[0996] The server analyzes the user's remote meeting schedule and instructs the "environmental adjustment device" on the optimal environment settings based on that.

[0997] Input: Remote meeting schedule information, emotion data

[0998] Data processing: Analyze meeting time and user sentiment to determine appropriate environment settings

[0999] Output: Adjusted lighting, projection settings, and sound settings

[1000] Specific operation: When a user inputs a remote meeting schedule into the device, the server analyzes the information and sends instructions to the "environmental adjustment device" to brighten the lights and play soothing music in the background before the meeting begins.

[1001] These processing steps enable the system to optimize the environment based on the user's real-time condition, providing a comfortable and efficient living environment.

[1002] (Application example 2)

[1003] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1004] While conventional environmental control systems can adjust the environment based on a user's behavioral patterns and the external environment, they lack the ability to dynamically adjust the environment based on the user's emotional state. Furthermore, they lack the ability to link with content distribution services, limiting the personalization of the user experience. The present invention aims to address these issues by integrating content distribution with highly accurate environmental adjustments based on the user's emotions.

[1005] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1006] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, and means for recognizing the user's emotions and automatically setting the optimal environment during content delivery, thereby enabling highly personalized environmental adjustment according to the user's emotional state and recommendation and delivery of content based on emotions.

[1007] "User behavior patterns" refers to records of the user's daily activities and behaviors and their tendencies.

[1008] "Preferences" refer to the preferences or tastes that a user has for a particular condition or option.

[1009] An "emotion engine" is a system for recognizing and analyzing emotions from a user's facial expressions, voice, and text data.

[1010] "Lighting" refers to electrical equipment and its functions for adjusting the intensity and color tone of light in a room or space.

[1011] "Temperature" is a parameter for measuring and controlling the hotness or coldness of air and objects within a particular space.

[1012] "Music" is an art form expressed through the combination of sounds, and includes elements such as rhythm, melody, and harmony.

[1013] "Video" means a moving or static visual representation that is displayed on a screen or display.

[1014] "Outside temperature" refers to the temperature of the environment outside the building.

[1015] "Interactive entertainment" refers to entertainment content that users can enjoy through two-way interaction with the system.

[1016] "Content distribution" refers to the provision of media content such as video and music to users over a digital network such as the Internet.

[1017] A "recommendation engine" is a system that suggests appropriate content based on a user's past behavior, preferences, and emotions.

[1018] To implement this invention, it is necessary to build a system that includes the following main elements:

[1019] System configuration

[1020] The system includes the following elements:

[1021] 1. Server: This is the central control unit that analyzes user behavior patterns, preferences, and emotions, and manages and adjusts various settings. The server analyzes data from user profiles and emotion engines, and issues instructions for appropriate environment settings and content delivery.

[1022] 2. Terminal: A device operated by the user (e.g., smartphone, tablet) that communicates with the server to send and receive user profile and setting information. The user's emotional state is also acquired from this device.

[1023] 3. Emotion Engine: An analysis device that recognizes emotions from the user's facial expressions, voice, text data, etc. and reflects them in the overall system settings. It uses OpenCV and dlib, Google Speech-to-Text API for voice analysis, and natural language processing (NLP) models for text analysis.

[1024] 4. Environmental control devices: These are devices that adjust the physical environment, such as lighting, thermostats, and smart speakers. They are controlled using the Philips Hue API or IFTTT.

[1025] System Operation

[1026] User Profile Settings

[1027] The server connects to the environmental control device when the system starts up and obtains the device's status and setting information. The terminal communicates with the server and reads the user profile, and the system prepares to set the environment based on the user's individual information.

[1028] Emotion-based environmental adjustment

[1029] The server analyzes the user's emotional data obtained from the emotion engine and, based on the results, issues instructions to adjust the environment, such as lighting, temperature, music, and video. For example, if the user is feeling stressed, the server will change the lighting to a softer color and play relaxing music.

[1030] Emotion-based content recommendation

[1031] The server uses a recommendation engine to recommend appropriate content (movies, music, images, etc.) based on the user's emotional state. For this purpose, it uses the Netflix Recommendation Engine and Spotify API.

[1032] Collaboration with environmental devices

[1033] The server manages the environmental control devices using the Philips Hue API and IFTTT, enabling dynamic environmental adjustments based on the user's emotions and behavioral patterns.

[1034] Specific examples

[1035] 1. Emotion recognition: The emotion engine analyzes the user's facial expression data and recognizes that the user is feeling "stressed."

[1036] 2. Content recommendation: The recommendation engine recommends relaxing music and movies to reduce stress.

[1037] 3. Environmental adjustment: The environmental control device changes the room lighting to a warmer color and sets the temperature to a comfortable level.

[1038] Example prompt sentence:

[1039] Describe a system that adjusts to a user's emotional state and environmental settings. If the user is stressed, the system selects comfortable lighting, relaxing music, and video streaming recommendations to soothe the user.

[1040] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1041] Step 1:

[1042] When the system starts up, the server connects to the environmental control device and obtains the device's status and setting information. The input information is the device's current status and settings, and the output information is the operating status and setting value of each device. Specifically, the server uses the Philips Hue API to obtain the current color tone and brightness of the lighting device, and checks the status of the thermostat via IFTTT.

[1043] Step 2:

[1044] The device communicates with the server and reads the user profile. The input information is the user's individual settings and past behavioral patterns, and the output information is the user profile data sent to the server. Specifically, the device collects emotional data and behavioral history from the user's smartphone and sends this to the server.

[1045] Step 3:

[1046] The server analyzes the user's emotional data obtained from the emotion engine. The input is the user's facial expression, voice, and text data, and the output is the emotion recognition results. Specifically, facial expression analysis is performed using "OpenCV" and "dlib," voice analysis is performed using "Google Speech-to-Text API," and the data is comprehensively analyzed using a "natural language processing (NLP) model" to determine the user's emotional state.

[1047] Step 4:

[1048] The server uses the emotion engine's analysis results to recommend appropriate content using a recommendation engine. The input is the emotion recognition results, and the output is a list of recommended content. Specifically, it uses the Netflix Recommendation Engine and Spotify API to recommend relaxing movies and music to reduce stress.

[1049] Step 5:

[1050] The server instructs the environmental control devices on optimal settings based on the analysis results of the emotion engine and the recommendations of the recommendation engine. The input is the emotional state and recommended content, and the output is specific setting instructions for each device. Specifically, it uses the Philips Hue API to change the color tone of the lighting to a warmer color and to set the thermostat to a comfortable temperature.

[1051] Step 6:

[1052] The device presents the settings and recommended content from the server to the user. The input is the settings and recommended content list from the server, and the output is data in a display format that the user can see and operate. Specifically, the settings and recommended movies and music are displayed in a pop-up on the smartphone's UI, allowing the user to select.

[1053] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1054] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1055] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1056] [Third embodiment]

[1057] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1058] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1059] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1060] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1061] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1062] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1063] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1064] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1065] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1066] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1067] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1068] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1069] The present invention relates to a system for making a user's living space more interactive and comfortable. Specific embodiments of the present invention will be described below.

[1070] System configuration

[1071] The system includes the following main elements:

[1072] 1. Server: A central control unit that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[1073] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[1074] 3. Lighting, thermostats, smart speakers, etc.: Devices for adjusting the physical environment, including integrated devices such as the "Smart Ceiling Nexus."

[1075] System Operation

[1076] Preparing the interface

[1077] When the system starts up, the server connects to "SmartCeiling Nexus" and obtains device status and configuration information. The device (smartphone or tablet) connects to the server via the application, and the user profile is loaded. This prepares the system to configure settings based on the user's individual information.

[1078] Providing a personalized experience

[1079] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes the user profile and determines appropriate settings based on the results, which are then sent to the device and "SmartCeiling Nexus."

[1080] Example: If a user prefers dim lighting when watching movies, the server can automatically dim the lighting when the movie is playing.

[1081] Predictive Comfort Management

[1082] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[1083] Example: If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[1084] Interactive Entertainment

[1085] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[1086] Example: Automatically set the lighting to red and the sound effects to emphasize the climax scene of a horror movie.

[1087] Optimizing energy efficiency

[1088] The server monitors the usage of each device and optimizes energy consumption by issuing instructions to the SmartCeiling Nexus to turn off unnecessary devices and functions, minimizing energy consumption.

[1089] Example: Turning off the lights automatically after a certain time has passed after the user leaves the room.

[1090] Health and Wellness Support

[1091] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and configures it to provide an appropriate environment.

[1092] Example: Automatically change the lighting to warmer colors and adjust the temperature before the user goes to sleep.

[1093] Interactive wall / ceiling games

[1094] The server analyzes the camera data, recognizes the user's movements, and sends interactive game content in real time to the "SmartCeiling Nexus," which projects it onto the walls and ceiling.

[1095] Example: When a user plays a jogging game, the course is projected onto a wall and the environment changes as the user moves.

[1096] Environment-based visual experiences

[1097] The server obtains data on the video content being played on the device, determines the optimal environmental settings (lighting, temperature, music), and issues instructions to "SmartCeiling Nexus."

[1098] Example: When playing a yoga video, adjust the lighting to a softer color and set the temperature appropriately to enhance the relaxation effect.

[1099] Optimizing remote work

[1100] The server analyzes the user's remote meeting schedule and provides the optimal environment settings. "SmartCeiling Nexus" starts the projector and adjusts the lighting and sound for the remote meeting.

[1101] Example: Before a remote meeting begins, start the projector and set the lighting appropriately to ensure a smooth start to the meeting.

[1102] These functions allow users to enjoy a comfortable and efficient living environment. The system of the present invention adjusts the environment interactively and automatically to provide an optimal environment according to the user's needs.

[1103] The processing flow will be explained below.

[1104] Step 1:

[1105] The server connects to the SmartCeiling Nexus when the system starts up and obtains device status and configuration information.

[1106] Step 2:

[1107] The terminal starts a dedicated application and transmits user authentication information to the server.

[1108] Step 3:

[1109] After verifying the authentication information, the server reads the user profile and analyzes data on past behavioral patterns and preferences.

[1110] Step 4:

[1111] The server determines personalized settings such as lighting, temperature, music, and video based on the analysis results.

[1112] Step 5:

[1113] The setting information determined by the server is sent to the terminal and "SmartCeiling Nexus."

[1114] Step 6:

[1115] "SmartCeiling Nexus" configures the physical environment, such as adjusting the color temperature of lighting, based on the setting information it receives.

[1116] Step 7:

[1117] The server collects real-time external weather data and room usage patterns to predictively determine a comfortable room temperature.

[1118] Step 8:

[1119] Based on instructions from the server, "SmartCeiling Nexus" automatically adjusts the heating and cooling settings.

[1120] Step 9:

[1121] When entertainment content is played on the terminal, the playback information is sent to the server.

[1122] Step 10:

[1123] The server sends instructions to the SmartCeiling Nexus to dynamically change lighting and sound settings based on the type of content and user preferences.

[1124] Step 11:

[1125] The SmartCeiling Nexus adjusts lighting and sound to provide an interactive entertainment experience.

[1126] Step 12:

[1127] The server monitors the energy usage of each device and issues instructions to turn off unnecessary devices.

[1128] Step 13:

[1129] "SmartCeiling Nexus" follows instructions from the server and automatically changes settings to optimize energy efficiency (e.g., turning off lights that have not been used for a long time).

[1130] Step 14:

[1131] The user's sleep and activity data collected on the device is sent to a server.

[1132] Step 15:

[1133] The server analyzes this data and determines settings (e.g., lighting, temperature) to provide an optimal sleeping environment.

[1134] Step 16:

[1135] The server sends instructions to the SmartCeiling Nexus, which adjusts the color temperature of the lighting and room temperature to create an optimal sleeping environment.

[1136] Step 17:

[1137] The server detects the start of an interactive game and analyzes real-time video data from the camera.

[1138] Step 18:

[1139] Game content is sent from the server to the "SmartCeiling Nexus" in real time and projected onto the walls and ceiling.

[1140] Step 19:

[1141] Data of the video content to be played on the terminal is transmitted to the server.

[1142] Step 20:

[1143] The server determines the optimal environmental settings for the video and sends that setting information to "SmartCeiling Nexus."

[1144] Step 21:

[1145] SmartCeiling Nexus adjusts lighting, temperature and music based on user instructions to provide an environment for relaxation and improved concentration.

[1146] Step 22:

[1147] The server analyzes the user's calendar information and checks for remote meeting schedules.

[1148] Step 23:

[1149] The server determines the optimal environment settings for the remote meeting and sends instructions to the SmartCeiling Nexus.

[1150] Step 24:

[1151] SmartCeiling Nexus activates the projector and adjusts lighting and sound for remote meetings.

[1152] Each of these processing steps taken allows the system to provide an interactive and comfortable living space for the user.

[1153] Example 1

[1154] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1155] Modern living environments require multiple devices and systems to enhance user comfort and efficiency, but these often operate independently, making integrated management difficult. Furthermore, meeting a wide range of needs, such as optimizing energy consumption, supporting health, and providing interactive entertainment, requires a high degree of automation and precise configuration based on user profiles. Conventional systems struggle to meet these requirements and are unable to provide a fully comfortable living environment for users.

[1156] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1157] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, means for acquiring device status and setting information at system startup, means for reading a user profile and setting the settings based on the individual information, and means for monitoring the user's device usage and optimizing energy efficiency to minimize energy consumption at all times. This makes it possible to manage and improve the user's comfort, health, energy efficiency, and entertainment experience in an integrated manner.

[1158] A "behavioral pattern" refers to the actions or choices a user makes repeatedly in a particular environment or situation.

[1159] "Preferences" refer to the preferences or tendencies of a user that are particularly favored in a particular setting or environment.

[1160] "Lighting" refers to artificial light sources in a user's living space.

[1161] "Temperature" refers to the indoor temperature in the user's living space.

[1162] "Music" refers to audio content that is played in the user's living space.

[1163] "Video" refers to video and image content that is visually displayed in the user's living space.

[1164] "Outside temperature" refers to the temperature of the air outside the living space.

[1165] "Predictive adjustment" means predicting future changes based on past data and current conditions, and making appropriate settings in advance.

[1166] "Reaction" refers to the behavior or response a user exhibits in response to a particular stimulus or situation.

[1167] "Selection" refers to a user selecting a specific item or setting from multiple options.

[1168] "Interactive entertainment" refers to entertainment content that changes dynamically in response to user input and actions.

[1169] "Boot up" refers to getting a system or device up and running and ready for use.

[1170] "Device" refers to electronic devices and machines used in the user's living space.

[1171] "Configuration information" refers to the operating conditions and parameters of a device or system.

[1172] A "user profile" refers to a data set that includes a user's personal information, preferences, behavioral patterns, etc.

[1173] "Individual information" refers to information or data that is unique to a particular user.

[1174] "Monitoring" refers to the continuous watching and recording of the operating status of a particular device or system.

[1175] "Energy efficiency" refers to the efficiency with which a particular device or system uses the energy it consumes.

[1176] The present invention relates to a system that makes a user's living space more interactive and comfortable. This system is mainly composed of a server, a terminal, and various devices (lighting, temperature control devices, smart speakers, etc.). An embodiment of the present invention will now be described in detail.

[1177] System configuration

[1178] The system includes the following main elements:

[1179] 1. Server: A central control unit that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[1180] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[1181] 3. Various devices: Devices for adjusting the physical environment, including "integrated devices" and "sensor clusters."

[1182] System Operation

[1183] Preparing the interface

[1184] When the system starts up, the server connects to the "integrated devices" and obtains the status and setting information of each device. This determines the initial state of the entire system. The device (smartphone or tablet) connects to the server via an application, and the user profile is loaded. This prepares the system to perform settings based on the user's individual information.

[1185] Example: When the system starts up, it receives information about the current room temperature and lighting settings from the integrated device. When you start the app on your smartphone and log in, the server retrieves the user's past activity log and settings.

[1186] Providing a personalized experience

[1187] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes the user profile and determines appropriate settings based on the results, which are then sent to the terminal and each device.

[1188] Example: When a user returns home, the living room lights automatically change to a warmer color and the settings are changed to play their favorite music.

[1189] Predictive Comfort Management

[1190] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[1191] Example: If the outside temperature is predicted to be high, the air conditioner will automatically turn on and adjust the set temperature appropriately.

[1192] Device usage monitoring and energy management

[1193] The server monitors the usage of each device and optimizes energy consumption, issuing instructions to turn off unnecessary devices and functions.

[1194] Example: Lights turn off automatically after a certain time has passed since the last time the user operated them.

[1195] Health data analysis and response

[1196] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and sets optimal settings.

[1197] Example: When the user goes to sleep, the lights change to warm colors to automatically create a comfortable sleeping environment.

[1198] Executing interactive functions

[1199] The server analyzes the camera data, recognizes the user's movements, and sends interactive game content in real time to the "integrated device," which projects it onto the walls and ceiling.

[1200] Example: Dynamic graphics are projected onto the wall as a user plays an exercise game with their family.

[1201] Preferences based on video content

[1202] The server obtains data on the video content being played on the terminal, determines the optimal environment settings, and issues instructions to each device.

[1203] Example: When you play a yoga video, the lighting changes to a softer, more relaxing color and the room temperature adjusts appropriately.

[1204] Optimizing the remote work environment

[1205] The server analyzes the user's remote conference schedule and provides optimal environment settings, while the integrated device activates the projector and adjusts the lighting and sound for the remote conference.

[1206] Example: The projector automatically starts 15 minutes before the start of a meeting, and the brightness and sound settings are adjusted to suit the meeting.

[1207] With these functions, users can enjoy a comfortable and efficient living environment. The system adjusts the environment interactively and automatically to provide an optimal environment according to the user's needs.

[1208] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1209] Step 1: Boot up the system and connect devices

[1210] When the system starts up, the server connects to the integrated devices and obtains the status and setting information of each device. This determines the initial state of the entire system. The input is the system startup signal and the response from the integrated devices, and the output is the current status and setting information of each device.

[1211] How it works: When the system starts up, the server sends a signal to the integrated device and receives information about the current room temperature and lighting settings.

[1212] Step 2: Loading the User Profile

[1213] The device (smartphone or tablet) connects to the server via an application and reads the user profile. The input is the device's login information, and the output is profile information that includes the user's past behavioral data and preferences.

[1214] Specific operation: When a user launches the app on their smartphone and logs in, the server retrieves the user's past activity logs and settings.

[1215] Step 3: Apply personalized settings

[1216] The server analyzes the user profile and automatically adjusts lighting, temperature, music, video, etc. based on past behavioral patterns and preferences. The input is the user profile and current environmental data, and the output is automatically adjusted setting instructions.

[1217] Specific operation: When the user returns home, the living room lights will automatically change to a warm color and the settings will change to play their favorite music.

[1218] Step 4: Real-time environmental adjustment

[1219] The server analyzes external weather data and room usage patterns in real time and predictively adjusts the room temperature accordingly. The input is weather data and usage pattern data, and the output is room temperature adjustment instructions.

[1220] Specific operation: If the outside temperature is predicted to be high, the air conditioner will automatically turn on and adjust the set temperature appropriately.

[1221] Step 5: Monitor device usage and manage energy

[1222] The server monitors the usage of each device and optimizes energy consumption. The input is the usage data of each device, and the output is instructions to turn off unnecessary devices and functions.

[1223] Specific operation: The light will automatically turn off after a certain period of time has passed since the user last operated the light.

[1224] Step 6: Analyze and respond to health data

[1225] The device collects the user's sleep and activity data and sends it to a server. The input is the user's health data, and the output is the optimal environmental settings.

[1226] Specific operation: Before the user goes to sleep, the lighting changes to a warm color, automatically creating a comfortable sleeping environment.

[1227] Step 7: Execute Interactive Features

[1228] The server analyzes the camera data and recognizes the user's movements. The input is the camera image data, and the output is instructions for interactive game content.

[1229] Specific Action: Dynamic graphics are projected onto the wall as the user plays an exercise game with their family.

[1230] Step 8: Setting preferences based on video content

[1231] The server obtains the data of the video content being played on the device and determines the optimal environment settings. The input is the video content data, and the output is instructions for the environment settings.

[1232] What it does: When you play a yoga video, the lighting changes to a soft, relaxing color and the room temperature is adjusted appropriately.

[1233] Step 9: Optimize your remote work environment

[1234] The server analyzes the user's remote conference schedule and provides instructions for optimal environment settings. The input is the remote conference schedule data, and the output is instructions for environment settings suitable for the conference.

[1235] Specific operation: The projector automatically starts up 15 minutes before the start of the meeting, and the brightness and sound settings are adjusted appropriately for the meeting.

[1236] (Application example 1)

[1237] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1238] While conventional systems were able to make users' living spaces interactive, they still had challenges in personalizing the customer experience in physical stores and providing individual product information and navigation using augmented reality technology. Furthermore, it was difficult to automatically adjust the in-store environment based on the customer's profile, making it impossible to provide a comfortable shopping experience with appropriate environmental settings.

[1239] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1240] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, means for personalizing the in-store environment to suit the customer's preferences based on the store's customer profile, means for providing product information to customers using augmented reality technology, providing detailed information on related products and navigating the store, and means for setting the optimal environment based on the analysis of the customer's profile. This makes it possible to improve the customer experience in physical stores and provide the optimal shopping environment for each individual customer.

[1241] "User behavior patterns and preferences" refers to the past repeated behavior and personal preference settings of a particular user.

[1242] "Means for automatically adjusting lighting, temperature, music, and video" refers to a mechanism that automatically changes or sets related environmental elements (lighting, temperature, music, and video) based on the user's settings and behavioral patterns.

[1243] "Means for analyzing behavioral patterns and outside temperatures and predictively adjusting the room temperature to an appropriate level" refers to a function that analyzes user behavioral data and external weather information, predicts the ideal room temperature, and adjusts it in advance.

[1244] "Means for providing interactive entertainment in response to user responses and selections" refers to a function that provides entertainment that changes dynamically based on the user's real-time responses and selections.

[1245] "Means to personalize the in-store environment to suit customer preferences based on the store's customer profile" refers to a system that adjusts the in-store environment, including lighting, temperature, and music, based on the profile information of customers who visit the store.

[1246] "Means for providing customers with product information, detailed guidance on related products, and in-store navigation using augmented reality technology" refers to a method or device that uses augmented reality (AR) technology to display product information in the customer's field of vision, and supports guidance on related products and in-store navigation.

[1247] "Means for optimal environmental settings based on customer profile analysis" refers to a function that analyzes customer profile data and automatically adjusts ideal environmental settings (lighting, temperature, music, etc.) based on that data.

[1248] System configuration

[1249] A system for implementing this invention includes the following main elements:

[1250] 1. Server: This is the central control device that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[1251] 2. Terminal: A device operated by the user (smartphone, tablet, smart glasses) that communicates with the server and sends and receives user profile and configuration information.

[1252] 3. Devices: Lighting, temperature control, music playback, augmented reality devices, and other devices that adjust the physical environment and enhance the user's experience.

[1253] System Operation

[1254] Preparing the interface

[1255] The server connects to each device at system startup to obtain status and configuration information. The device connects to the server via an application, and the user profile is loaded. The system is then ready to configure itself based on the user's individual information.

[1256] Providing a personalized experience

[1257] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. It analyzes the user profile and determines the appropriate settings based on the results, sending them to the terminal and each device.

[1258] Examples:

[1259] If a customer prefers dim lighting when watching a movie, the server can automatically dim the lighting when the movie is playing.

[1260] Predictive Comfort Management

[1261] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[1262] Examples:

[1263] If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[1264] Interactive Entertainment

[1265] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[1266] Examples:

[1267] During the climax of a horror movie, the lighting is automatically set to red and the sound effects are emphasized.

[1268] Personalized environment based on store customer profiles

[1269] The server analyzes the profiles of customers who visit the store and personalizes the in-store environment to suit their preferences, allowing customers to enjoy the optimal shopping environment.

[1270] Examples:

[1271] By playing music that customers like and adjusting lighting and temperature appropriately, we provide a comfortable shopping experience.

[1272] Augmented reality technology for product information and navigation

[1273] The device uses augmented reality technology to display product information to the customer's visual field, providing detailed information on related products and navigation.

[1274] Examples:

[1275] As customers walk through the store using the smart glasses, a guide displayed on the glasses navigates them to the products they want to purchase and displays product features and pricing information.

[1276] Prompt Sentence Examples

[1277] "Automatically adjust the in-store environment (lighting, music, temperature) based on the user's profile. Also, if a customer is wearing smart glasses, show them detailed information about related products and help them navigate to the products they are looking for."

[1278] In this way, the system of the present invention can interactively and automatically adjust environmental settings and entertainment based on user behavior and preferences to enhance the in-store customer experience.

[1279] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1280] Step 1:

[1281] The server connects to each device at system startup and obtains status and configuration information.

[1282] Input: Identification and initial configuration data for each device.

[1283] Output: Current status and configuration information for each device.

[1284] The server collects the setting information sent by each device and adjusts the initial settings based on this.

[1285] Step 2:

[1286] The device connects to the server via the application and the user profile is loaded.

[1287] Input: User credentials and profile data.

[1288] Output: The user's individual profile.

[1289] The terminal launches the application and connects to the server to request and retrieve the user's profile.

[1290] Step 3:

[1291] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's behavioral patterns and preferences.

[1292] Input: User behavioral pattern data and preference settings.

[1293] Output: Adjustment instructions for each device.

[1294] The server analyzes the user's past behavioral patterns and preferences, and based on this calculates appropriate lighting, temperature, music, and video settings and sends them to the relevant devices.

[1295] Step 4:

[1296] The server analyzes external weather data and room usage patterns in real time and predictively adjusts the room temperature to an appropriate level.

[1297] Input: External weather data and room usage pattern data.

[1298] Output: Room temperature control command.

[1299] The server acquires external weather data, compares it with the user's room usage pattern, predicts the room temperature adjustment value, and sends it to the temperature adjustment device.

[1300] Step 5:

[1301] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[1302] Input: entertainment content data and user reaction data.

[1303] Output: Instructions to change lighting and sound settings.

[1304] The device plays entertainment, and the server collects user reactions in real time from sensor data and adjusts lighting and sound accordingly.

[1305] Step 6:

[1306] The server analyzes the profiles of customers who visit the store and personalizes the in-store environment to suit their preferences.

[1307] Input: Customer profile data.

[1308] Output: Adjustment instructions to each device in the store.

[1309] The server analyzes the profile information of customers visiting the store and issues instructions to each device in the store to adjust lighting, music, temperature, etc. based on the results.

[1310] Step 7:

[1311] The device uses augmented reality technology to display product information to the customer's visual field, providing detailed information on related products and navigation.

[1312] Input: Customer navigation request and product information data.

[1313] Output: Augmented reality display data.

[1314] The terminal (such as smart glasses) retrieves product information from the server based on the customer's request, displays it using augmented reality technology, and guides the customer to the location of the desired product.

[1315] This will improve the customer experience in physical stores and provide an optimal shopping environment for each individual customer.

[1316] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1317] The system of the present invention enables more advanced personalization by combining means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for predictively adjusting the room temperature by analyzing the outside temperature, and means for providing interactive entertainment, as well as an emotion engine that recognizes the user's emotions.

[1318] System configuration

[1319] The system includes the following main elements:

[1320] 1. Server: The central control unit that analyzes user behavior patterns, preferences, and emotions, and manages and adjusts various settings.

[1321] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[1322] 3. Emotion engine: An analysis device that recognizes emotions from the user's facial expressions, voice, text data, etc. and reflects them in the settings of the entire system.

[1323] 4. Lighting, thermostats, smart speakers, etc.: Devices for adjusting the physical environment, including integrated devices such as the "Smart Ceiling Nexus."

[1324] System Operation

[1325] Preparing the interface

[1326] When the system starts up, the server connects to "SmartCeiling Nexus" to obtain device status and configuration information. The device (smartphone or tablet) connects to the server via the application, and the user profile is loaded. This prepares the system to configure settings based on the user's individual information.

[1327] Providing a personalized experience

[1328] The server determines personalized settings for lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes data from the user profile and emotion engine, determines appropriate settings based on the results, and sends them to the device and "SmartCeiling Nexus."

[1329] Example: If the emotion engine detects that the user is feeling stressed, the server will set soft lighting and calming music to enhance relaxation.

[1330] Predictive Comfort Management

[1331] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[1332] Example: If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[1333] Interactive Entertainment

[1334] As entertainment content is played on the terminal, the server analyzes the user's responses, preferences and emotions in real time and dynamically changes lighting and sound settings.

[1335] Example: During the climax of a horror movie, if the user is recognized as feeling scared, the lighting will be automatically set to red and the sound effects will be enhanced.

[1336] Optimizing energy efficiency

[1337] The server monitors the usage of each device and optimizes energy consumption by issuing instructions to the SmartCeiling Nexus to turn off unnecessary devices and functions, minimizing energy consumption.

[1338] Example: Turning off the lights automatically after a certain time has passed after the user leaves the room.

[1339] Health and Wellness Support

[1340] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and configures it to provide an appropriate environment.

[1341] Example: Before bedtime, if the emotion engine detects that the user is in a state of relaxation, it will change the lighting to a warmer color and keep the temperature optimal.

[1342] Interactive wall / ceiling games

[1343] The server analyzes the camera data and works in conjunction with the emotion engine to recognize the user's movements and emotions. Based on this, interactive game content is sent in real time to the "SmartCeiling Nexus" and projected onto the walls and ceiling.

[1344] Example: When a user plays a jogging game, if the emotion engine recognizes that the user is having fun, it will randomly add fun elements (light and sound effects).

[1345] Environment-based visual experiences

[1346] The server obtains data on the video content being played on the device, determines the optimal environmental settings (lighting, temperature, music), and issues instructions to "SmartCeiling Nexus."

[1347] Example: When playing a yoga video, if the emotion engine detects a strong sense of relaxation, it will soften the lighting and set the temperature optimally.

[1348] Optimizing remote work

[1349] The server analyzes the user's remote meeting schedule and provides the optimal environment settings. "SmartCeiling Nexus" starts the projector and adjusts the lighting and sound for the remote meeting.

[1350] Example: Before a remote meeting begins, if the emotion engine detects stress, it will turn the lights up brighter and play calming music in the background.

[1351] These functions allow users to enjoy a comfortable and efficient living environment. The system of the present invention adjusts the environment interactively and automatically, providing an optimal environment according to the user's needs and emotions.

[1352] The processing flow will be explained below.

[1353] Step 1:

[1354] The server connects to the SmartCeiling Nexus when the system starts up and obtains device status and configuration information.

[1355] Step 2:

[1356] The terminal starts a dedicated application and transmits user authentication information to the server.

[1357] Step 3:

[1358] After verifying the authentication information, the server reads the user profile and analyzes data on past behavioral patterns and preferences.

[1359] Step 4:

[1360] The emotion engine analyzes the user's facial expressions and voice and generates emotion data.

[1361] Step 5:

[1362] The server determines personalized settings such as lighting, temperature, music, and video based on the analysis results and emotional data.

[1363] Step 6:

[1364] The setting information determined by the server is sent to the terminal and "SmartCeiling Nexus."

[1365] Step 7:

[1366] "SmartCeiling Nexus" configures the physical environment, such as adjusting the color temperature of lighting, based on the setting information it receives.

[1367] Step 8:

[1368] The server collects real-time external weather data and room usage patterns to predictively determine a comfortable room temperature.

[1369] Step 9:

[1370] Based on instructions from the server, "SmartCeiling Nexus" automatically adjusts the heating and cooling settings.

[1371] Step 10:

[1372] When entertainment content is played on the terminal, the playback information is sent to the server.

[1373] Step 11:

[1374] The emotion engine analyzes users' reactions in real time while watching entertainment and generates emotion data.

[1375] Step 12:

[1376] The server sends instructions to the SmartCeiling Nexus to dynamically change lighting and sound settings based on the type of content and emotional data.

[1377] Step 13:

[1378] The SmartCeiling Nexus adjusts lighting and sound to provide an interactive entertainment experience.

[1379] Step 14:

[1380] The server monitors the energy usage of each device and issues instructions to turn off unnecessary devices.

[1381] Step 15:

[1382] "SmartCeiling Nexus" follows instructions from the server and automatically changes settings to optimize energy efficiency (e.g., turning off lights that have not been used for a long time).

[1383] Step 16:

[1384] The user's sleep and activity data collected on the device is sent to a server.

[1385] Step 17:

[1386] The emotion engine analyzes the user's emotions before sleep and checks their state of relaxation.

[1387] Step 18:

[1388] The server analyzes this data and emotional data to determine settings (e.g., lighting, temperature) to provide an optimal sleeping environment.

[1389] Step 19:

[1390] The server sends instructions to the SmartCeiling Nexus, which adjusts the color temperature of the lighting and room temperature to create an optimal sleeping environment.

[1391] Step 20:

[1392] The server detects the start of an interactive game and analyzes real-time video data from the camera.

[1393] Step 21:

[1394] The emotion engine analyzes the user's emotions during the game in real time and generates emotion data.

[1395] Step 22:

[1396] The server transmits the game content to the "SmartCeiling Nexus" in real time, where it is projected onto the walls and ceiling.

[1397] Step 23:

[1398] Data of the video content to be played on the terminal is transmitted to the server.

[1399] Step 24:

[1400] The emotion engine analyzes the user's emotions while watching the video and confirms that they are relaxed.

[1401] Step 25:

[1402] The server determines the optimal environmental settings for the video and sends that setting information to "SmartCeiling Nexus."

[1403] Step 26:

[1404] SmartCeiling Nexus adjusts lighting, temperature and music based on user instructions to provide an environment for relaxation and improved concentration.

[1405] Step 27:

[1406] The server analyzes the user's calendar information and checks for remote meeting schedules.

[1407] Step 28:

[1408] The emotion engine analyzes the user's emotions before a remote meeting and confirms that they are feeling stressed.

[1409] Step 29:

[1410] The server determines the optimal environment settings for the remote meeting and sends instructions to the SmartCeiling Nexus.

[1411] Step 30:

[1412] SmartCeiling Nexus activates the projector and adjusts lighting and sound for remote meetings.

[1413] Each of these processing steps taken allows the system to provide an interactive and comfortable living space for the user.

[1414] Example 2

[1415] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1416] Conventional environmental control systems have difficulty in fine-tuning the environment based on the user's behavioral patterns, preferences, and emotions, and are therefore inadequate in optimizing the environment to suit the user's condition in real time. Furthermore, they do not take into account efficient energy use, and are unable to provide interactive entertainment that reflects the user's health and emotions.

[1417] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for analyzing the behavioral patterns and the outside temperature and predictively adjusting the room temperature to an appropriate level, means for recognizing the user's emotions and providing personalized settings according to the emotions, means for providing interactive entertainment according to the user's reactions and selections, and means for monitoring energy consumption and optimizing energy efficiency by turning off unnecessary devices. This makes it possible to optimize the environment based on the user's real-time state, improve energy efficiency, and provide interactive entertainment according to the user's emotions and health state.

[1418] "User behavior patterns" refer to the behavioral tendencies and routines that a user takes in their daily life, such as the time they return home, the time they go to bed, and the devices they use.

[1419] "Preferences" refer to personal tastes and preferences that a user has for a particular environmental setting, such as a favorite genre of music or a preferred lighting brightness.

[1420] "Lighting" refers to an artificial light source used to brighten a room or space, and whose brightness, color, flashing pattern, etc. can be adjusted.

[1421] "Temperature" refers to the temperature of a room or space, and can be adjusted using air conditioning, heating, cooling, etc.

[1422] "Music" is a combination of sounds reproduced from a sound source, and is selected and reproduced based on the user's preferences.

[1423] "Video" refers to visual content that is presented to users through devices such as televisions, projectors, and monitors.

[1424] "Outside temperature" refers to the temperature of the air outside, and is obtained using weather forecasts and real-time weather data.

[1425] "Emotion" refers to the user's state of mind, such as feelings of joy, sadness, anger, surprise, etc.

[1426] "Emotion recognition" means using sensors and analysis engines to identify emotions from a user's facial expressions, tone of voice, text input, etc.

[1427] "Personalized settings" refers to environmental settings that match a user's specific preferences, behavioral patterns, and emotions, and includes lighting, music, temperature, and video settings that are optimized for each individual user.

[1428] "Real-time monitoring" refers to monitoring that occurs at the present moment, and involves the immediate acquisition and analysis of data from sensors and devices.

[1429] "Optimizing energy efficiency" means minimizing energy consumption and using resources in a sustainable manner.

[1430] "Interactive entertainment" refers to entertainment content that changes depending on the user's reactions and choices, allowing for interactive enjoyment.

[1431] The system of the present invention automatically adjusts environmental settings based on a user's behavioral patterns and preferences, and further provides personalized settings by recognizing the user's emotions in real time using an emotion engine. The system includes the following main components:

[1432] 1. Server:

[1433] The server is the central control device of the system, analyzing user behavior patterns and emotional data to optimize environmental settings. The server is connected to an integrated device called "SmartCeiling Nexus" and acquires device status and setting information. The server dynamically changes environmental settings such as lighting, temperature, music, and video using user profiles and real-time sensor data.

[1434] 2. Terminal:

[1435] The terminal (smartphone or tablet) is the device operated by the user and is responsible for communicating with the server. The user logs in to the system using the terminal and sends their profile and settings information to the server, which allows the server to understand the user's preferences and behavioral patterns.

[1436] 3. Emotion Engine:

[1437] The emotion engine is a device that analyzes the user's facial expressions, voice, text data, etc. to recognize emotions. The emotion engine uses sensors such as cameras and microphones to collect real-time data and analyzes the user's current emotional state based on that data. The analysis results are sent to a server and used to optimize environment settings.

[1438] 4. Environmental control device:

[1439] Environmental control devices such as lighting, air conditioning, and smart speakers are integrated into the "SmartCeiling Nexus," which adjusts the physical environment based on instructions from the server.

[1440] To illustrate, consider the following scenario:

[1441] Scenario 1: Adjusting your environment when you get home

[1442] When a user returns home from work, the device connects to the home Wi-Fi and sends a notification to the server. The server reads the user's profile and analyzes the data obtained from the emotion engine. For example, if the user is feeling stressed, the server instructs the SmartCeiling Nexus to set soft lighting and relaxing music.

[1443] Example prompt sentence:

[1444] "Users are coming home and feeling stressed, so set up soft lighting and calming music to promote relaxation."

[1445] Scenario 2: Entertainment optimization

[1446] When a user watches a horror movie, the device sends playback information to the server. The server uses data from the emotion engine to analyze the user's reaction in real time. For example, if the user is feeling scared, the server instructs SmartCeiling Nexus to dim the lights and enhance the sound effects.

[1447] Example prompt sentence:

[1448] "The user is feeling scared during the climax of a horror movie, so dim the lights and amplify the sound effects."

[1449] Scenario 3: Health Support

[1450] Before the user goes to sleep, activity data is sent from the smartwatch to the server via the device. The server then uses this data and the analysis results of the emotion engine to provide an optimal sleeping environment, for example by turning the lighting to a warm color and setting the room temperature to a comfortable temperature.

[1451] Example prompt sentence:

[1452] "Because users are experiencing relaxation before bed, change the lighting to warmer colors and set the room temperature optimally."

[1453] This system can provide an optimal environment according to the user's real-time condition and improve energy efficiency. In particular, the introduction of an emotion engine enables fine-tuned adjustments to match the user's emotions, which is expected to significantly improve quality of life.

[1454] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1455] Step 1: Initialization and Connection

[1456] When the system starts up, the server first connects to the "environmental adjustment device" (e.g., "SmartCeiling Nexus") to obtain device status and configuration information. This initialization procedure prepares the system.

[1457] Input: Device identification information, network settings

[1458] Data processing: Capture device status information and store it in a database on the server.

[1459] Output: Connection status log, current settings information

[1460] Specific operation: The server detects the "SmartCeiling Nexus" via the network, obtains the status of LED lighting, air conditioning, music playback devices, etc., and creates a log that stores the status.

[1461] Step 2: Loading the User Profile

[1462] The terminal transmits the user's login information to the server, and the server retrieves the user profile and reads the user's preferences and behavioral patterns.

[1463] Input: User authentication information, User ID

[1464] Data processing: Reading and analyzing user profile information from the database

[1465] Output: User's past behavior patterns and preference settings

[1466] Specific operation: When a user logs in with a smartphone, the device sends the user ID and authentication information to the server, which then retrieves relevant information from the database and activates the user profile.

[1467] Step 3: Real-time data collection and emotion recognition

[1468] The emotion engine collects real-time data from sensors such as cameras and microphones, analyzes the user's emotions, and transmits this information to a server.

[1469] Input: Real-time video, audio, and text data

[1470] Data processing: Input data into the emotion analysis model to classify the emotional state

[1471] Output: User's emotional state (e.g., stressed, relaxed, happy)

[1472] Specific operation: When a user enters a room, the emotion engine collects data from the camera and microphone, performs emotion analysis in real time, and sends the results to the server.

[1473] Step 4: Optimize and adjust settings

[1474] The server optimizes specific environmental settings based on the user's past behavioral patterns, real-time data, and emotional state, and transmits the settings information to an "environment adjustment device."

[1475] Input: User profile, real-time emotional data, behavioral patterns, outside temperature data

[1476] Data processing: Comprehensive analysis of input data and calculation of optimal environment settings

[1477] Output: Optimized lighting, temperature, music, and video settings instructions

[1478] Specific operation: If the user is analyzed as feeling stressed, the server instructs the "environmental adjustment device" to set warm lighting and relaxing music.

[1479] Step 5: Sync your entertainment content

[1480] The device plays entertainment content and sends the information to the server, which then adjusts the environment settings based on the emotion data and content information.

[1481] Input: Entertainment content information currently being played, emotional data

[1482] Data processing: Analyzing the type of content and the user's emotional state to determine appropriate environment settings

[1483] Output: Adjusted lighting and sound settings

[1484] Specific operation: When a user watches a horror movie, the device sends playback information to the server, and the server analyzes the emotion engine data and sends instructions to the "environment adjustment device" to dim the lights and emphasize the sound effects.

[1485] Step 6: Monitor and optimize energy efficiency

[1486] The server monitors the usage of each device and optimizes energy efficiency by automatically turning off unnecessary devices.

[1487] Input: Operating status and energy consumption data of each device

[1488] Data processing: Analyze usage patterns and energy consumption to identify unnecessary devices

[1489] Output: Instructions to turn off devices, energy usage reports

[1490] Specific operation: After the user leaves the room, the server sends an instruction to the "environmental control device" to turn off all unnecessary lights and devices after a certain period of time.

[1491] Step 7: Monitor your health

[1492] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and provides appropriate settings.

[1493] Input: User's sleep data, activity data

[1494] Data processing: Analyze health data and determine optimal environmental settings

[1495] Output: Adjustable lighting and temperature settings

[1496] Specific operation: The device sends activity data obtained by the user from the smartwatch to the server, and the server adjusts the lighting and temperature based on that data.

[1497] Step 8: Controlling the Interactive Game

[1498] The server analyzes the camera data and works in conjunction with an emotion engine to recognize the user's movements and emotions, and provides interactive game content based on this.

[1499] Input: Real-time camera data, emotion data

[1500] Data processing: Analyzing user movements and emotions to adjust the game's appearance and behavior

[1501] Output: Instructions for changing game content, setting visual and sound effects

[1502] How it works: When a user plays an immersive game, the camera and emotion engine collect and analyze data, and based on the results, send instructions to the "environment adjustment device" to adjust effects and difficulty in real time.

[1503] Step 9: Delivering an environment-based visual experience

[1504] The server acquires data on the video content being played on the terminal, determines the optimal environmental settings, and issues instructions to the "environment adjustment device."

[1505] Input: Video content information, real-time emotion data

[1506] Data processing: Analyze content attributes and user sentiment to calculate optimal environment settings

[1507] Output: Adjusted lighting, temperature, music settings

[1508] How it works: When a user plays a yoga video, the server recognizes it and sends instructions to the "environmental control device" to adjust the lighting and temperature to enhance the relaxation effect.

[1509] Step 10: Optimize your remote work environment

[1510] The server analyzes the user's remote meeting schedule and instructs the "environmental adjustment device" on the optimal environment settings based on that.

[1511] Input: Remote meeting schedule information, emotion data

[1512] Data processing: Analyze meeting time and user sentiment to determine appropriate environment settings

[1513] Output: Adjusted lighting, projection settings, and sound settings

[1514] Specific operation: When a user inputs a remote meeting schedule into the device, the server analyzes the information and sends instructions to the "environmental adjustment device" to brighten the lights and play soothing music in the background before the meeting begins.

[1515] These processing steps enable the system to optimize the environment based on the user's real-time condition, providing a comfortable and efficient living environment.

[1516] (Application example 2)

[1517] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1518] While conventional environmental control systems can adjust the environment based on a user's behavioral patterns and the external environment, they lack the ability to dynamically adjust the environment based on the user's emotional state. Furthermore, they lack the ability to link with content distribution services, limiting the personalization of the user experience. The present invention aims to address these issues by integrating content distribution with highly accurate environmental adjustments based on the user's emotions.

[1519] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1520] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, and means for recognizing the user's emotions and automatically setting the optimal environment during content delivery, thereby enabling highly personalized environmental adjustment according to the user's emotional state and recommendation and delivery of content based on emotions.

[1521] "User behavior patterns" refers to records of the user's daily activities and behaviors and their tendencies.

[1522] "Preferences" refer to the preferences or tastes that a user has for a particular condition or option.

[1523] An "emotion engine" is a system for recognizing and analyzing emotions from a user's facial expressions, voice, and text data.

[1524] "Lighting" refers to electrical equipment and its functions for adjusting the intensity and color tone of light in a room or space.

[1525] "Temperature" is a parameter for measuring and controlling the hotness or coldness of air and objects within a particular space.

[1526] "Music" is an art form expressed through the combination of sounds, and includes elements such as rhythm, melody, and harmony.

[1527] "Video" means a moving or static visual representation that is displayed on a screen or display.

[1528] "Outside temperature" refers to the temperature of the environment outside the building.

[1529] "Interactive entertainment" refers to entertainment content that users can enjoy through two-way interaction with the system.

[1530] "Content distribution" refers to the provision of media content such as video and music to users over a digital network such as the Internet.

[1531] A "recommendation engine" is a system that suggests appropriate content based on a user's past behavior, preferences, and emotions.

[1532] To implement this invention, it is necessary to build a system that includes the following main elements:

[1533] System configuration

[1534] The system includes the following elements:

[1535] 1. Server: This is the central control unit that analyzes user behavior patterns, preferences, and emotions, and manages and adjusts various settings. The server analyzes data from user profiles and emotion engines, and issues instructions for appropriate environment settings and content delivery.

[1536] 2. Terminal: A device operated by the user (e.g., smartphone, tablet) that communicates with the server to send and receive user profile and setting information. The user's emotional state is also acquired from this device.

[1537] 3. Emotion Engine: An analysis device that recognizes emotions from the user's facial expressions, voice, text data, etc. and reflects them in the overall system settings. It uses OpenCV and dlib, Google Speech-to-Text API for voice analysis, and natural language processing (NLP) models for text analysis.

[1538] 4. Environmental control devices: These are devices that adjust the physical environment, such as lighting, thermostats, and smart speakers. They are controlled using the Philips Hue API or IFTTT.

[1539] System Operation

[1540] User Profile Settings

[1541] The server connects to the environmental control device when the system starts up and obtains the device's status and setting information. The terminal communicates with the server and reads the user profile, and the system prepares to set the environment based on the user's individual information.

[1542] Emotion-based environmental adjustment

[1543] The server analyzes the user's emotional data obtained from the emotion engine and, based on the results, issues instructions to adjust the environment, such as lighting, temperature, music, and video. For example, if the user is feeling stressed, the server will change the lighting to a softer color and play relaxing music.

[1544] Emotion-based content recommendation

[1545] The server uses a recommendation engine to recommend appropriate content (movies, music, images, etc.) based on the user's emotional state. For this purpose, it uses the Netflix Recommendation Engine and Spotify API.

[1546] Collaboration with environmental devices

[1547] The server manages the environmental control devices using the Philips Hue API and IFTTT, enabling dynamic environmental adjustments based on the user's emotions and behavioral patterns.

[1548] Specific examples

[1549] 1. Emotion recognition: The emotion engine analyzes the user's facial expression data and recognizes that the user is feeling "stressed."

[1550] 2. Content recommendation: The recommendation engine recommends relaxing music and movies to reduce stress.

[1551] 3. Environmental adjustment: The environmental control device changes the room lighting to a warmer color and sets the temperature to a comfortable level.

[1552] Example prompt sentence:

[1553] Describe a system that adjusts to a user's emotional state and environmental settings. If the user is stressed, the system selects comfortable lighting, relaxing music, and video recommendations to soothe the user.

[1554] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1555] Step 1:

[1556] When the system starts up, the server connects to the environmental control device and obtains the device's status and setting information. The input information is the device's current status and settings, and the output information is the operating status and setting value of each device. Specifically, the server uses the Philips Hue API to obtain the current color tone and brightness of the lighting device, and checks the status of the thermostat via IFTTT.

[1557] Step 2:

[1558] The device communicates with the server and reads the user profile. The input information is the user's individual settings and past behavioral patterns, and the output information is the user profile data sent to the server. Specifically, the device collects emotional data and behavioral history from the user's smartphone and sends this to the server.

[1559] Step 3:

[1560] The server analyzes the user's emotional data obtained from the emotion engine. The input is the user's facial expression, voice, and text data, and the output is the emotion recognition results. Specifically, facial expression analysis is performed using "OpenCV" and "dlib," voice analysis is performed using "Google Speech-to-Text API," and the data is comprehensively analyzed using a "natural language processing (NLP) model" to determine the user's emotional state.

[1561] Step 4:

[1562] The server uses the emotion engine's analysis results to recommend appropriate content using a recommendation engine. The input is the emotion recognition results, and the output is a list of recommended content. Specifically, it uses the Netflix Recommendation Engine and Spotify API to recommend relaxing movies and music to reduce stress.

[1563] Step 5:

[1564] The server instructs the environmental control devices on optimal settings based on the analysis results of the emotion engine and the recommendations of the recommendation engine. The input is the emotional state and recommended content, and the output is specific setting instructions for each device. Specifically, it uses the Philips Hue API to change the color tone of the lighting to a warmer color and to set the thermostat to a comfortable temperature.

[1565] Step 6:

[1566] The device presents the settings and recommended content from the server to the user. The input is the settings and recommended content list from the server, and the output is data in a display format that the user can see and operate. Specifically, the settings and recommended movies and music are displayed in a pop-up on the smartphone's UI, allowing the user to select.

[1567] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1568] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1569] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1570] [Fourth embodiment]

[1571] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1572] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1573] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1574] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1575] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1576] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1577] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1578] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1579] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1580] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1581] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1582] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1583] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1584] The present invention relates to a system for making a user's living space more interactive and comfortable. Specific embodiments of the present invention will be described below.

[1585] System configuration

[1586] The system includes the following main elements:

[1587] 1. Server: A central control unit that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[1588] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[1589] 3. Lighting, thermostats, smart speakers, etc.: Devices for adjusting the physical environment, including integrated devices such as the "Smart Ceiling Nexus."

[1590] System Operation

[1591] Preparing the interface

[1592] When the system starts up, the server connects to "SmartCeiling Nexus" and obtains device status and configuration information. The device (smartphone or tablet) connects to the server via the application, and the user profile is loaded. This prepares the system to configure settings based on the user's individual information.

[1593] Providing a personalized experience

[1594] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes the user profile and determines appropriate settings based on the results, which are then sent to the device and "SmartCeiling Nexus."

[1595] Example: If a user prefers dim lighting when watching movies, the server can automatically dim the lighting when the movie is playing.

[1596] Predictive Comfort Management

[1597] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[1598] Example: If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[1599] Interactive Entertainment

[1600] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[1601] Example: Automatically set the lighting to red and the sound effects to emphasize the climax scene of a horror movie.

[1602] Optimizing energy efficiency

[1603] The server monitors the usage of each device and optimizes energy consumption by issuing instructions to the SmartCeiling Nexus to turn off unnecessary devices and functions, minimizing energy consumption.

[1604] Example: Turning off the lights automatically after a certain time has passed after the user leaves the room.

[1605] Health and Wellness Support

[1606] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and configures it to provide an appropriate environment.

[1607] Example: Automatically change the lighting to warmer colors and adjust the temperature before the user goes to sleep.

[1608] Interactive wall / ceiling games

[1609] The server analyzes the camera data, recognizes the user's movements, and sends interactive game content in real time to the "SmartCeiling Nexus," which projects it onto the walls and ceiling.

[1610] Example: When a user plays a jogging game, the course is projected onto a wall and the environment changes as the user moves.

[1611] Environment-based visual experiences

[1612] The server obtains data on the video content being played on the device, determines the optimal environmental settings (lighting, temperature, music), and issues instructions to "SmartCeiling Nexus."

[1613] Example: When playing a yoga video, adjust the lighting to a softer color and set the temperature appropriately to enhance the relaxation effect.

[1614] Optimizing remote work

[1615] The server analyzes the user's remote meeting schedule and provides the optimal environment settings. "SmartCeiling Nexus" starts the projector and adjusts the lighting and sound for the remote meeting.

[1616] Example: Before a remote meeting begins, start the projector and set the lighting appropriately to ensure a smooth start to the meeting.

[1617] These functions allow users to enjoy a comfortable and efficient living environment. The system of the present invention adjusts the environment interactively and automatically to provide an optimal environment according to the user's needs.

[1618] The processing flow will be explained below.

[1619] Step 1:

[1620] The server connects to the SmartCeiling Nexus when the system starts up and obtains device status and configuration information.

[1621] Step 2:

[1622] The terminal starts a dedicated application and transmits user authentication information to the server.

[1623] Step 3:

[1624] After verifying the authentication information, the server reads the user profile and analyzes past behavioral patterns and preference data.

[1625] Step 4:

[1626] The server determines personalized settings such as lighting, temperature, music, and video based on the analysis results.

[1627] Step 5:

[1628] The setting information determined by the server is sent to the terminal and "SmartCeiling Nexus."

[1629] Step 6:

[1630] "SmartCeiling Nexus" configures the physical environment, such as adjusting the color temperature of lighting, based on the setting information it receives.

[1631] Step 7:

[1632] The server collects real-time external weather data and room usage patterns to predictively determine a comfortable room temperature.

[1633] Step 8:

[1634] Based on instructions from the server, "SmartCeiling Nexus" automatically adjusts the heating and cooling settings.

[1635] Step 9:

[1636] When entertainment content is played on the terminal, the playback information is sent to the server.

[1637] Step 10:

[1638] The server sends instructions to the SmartCeiling Nexus to dynamically change lighting and sound settings based on the type of content and user preferences.

[1639] Step 11:

[1640] The SmartCeiling Nexus adjusts lighting and sound to provide an interactive entertainment experience.

[1641] Step 12:

[1642] The server monitors the energy usage of each device and issues instructions to turn off unnecessary devices.

[1643] Step 13:

[1644] "SmartCeiling Nexus" follows instructions from the server and automatically changes settings to optimize energy efficiency (e.g., turning off lights that have not been used for a long time).

[1645] Step 14:

[1646] The user's sleep and activity data collected on the device is sent to a server.

[1647] Step 15:

[1648] The server analyzes this data and determines settings (e.g., lighting, temperature) to provide an optimal sleeping environment.

[1649] Step 16:

[1650] The server sends instructions to the SmartCeiling Nexus, which adjusts the color temperature of the lighting and room temperature to create an optimal sleeping environment.

[1651] Step 17:

[1652] The server detects the start of an interactive game and analyzes real-time video data from the camera.

[1653] Step 18:

[1654] Game content is sent from the server to the "SmartCeiling Nexus" in real time and projected onto the walls and ceiling.

[1655] Step 19:

[1656] Data of the video content to be played on the terminal is transmitted to the server.

[1657] Step 20:

[1658] The server determines the optimal environmental settings for the video and sends that setting information to "SmartCeiling Nexus."

[1659] Step 21:

[1660] SmartCeiling Nexus adjusts lighting, temperature and music based on user instructions to provide an environment for relaxation and improved concentration.

[1661] Step 22:

[1662] The server analyzes the user's calendar information and checks for remote meeting schedules.

[1663] Step 23:

[1664] The server determines the optimal environment settings for the remote meeting and sends instructions to the SmartCeiling Nexus.

[1665] Step 24:

[1666] SmartCeiling Nexus activates the projector and adjusts lighting and sound for remote meetings.

[1667] Each of these processing steps taken allows the system to provide an interactive and comfortable living space for the user.

[1668] Example 1

[1669] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1670] Modern living environments require multiple devices and systems to enhance user comfort and efficiency, but these often operate independently, making integrated management difficult. Furthermore, meeting a wide range of needs, such as optimizing energy consumption, supporting health, and providing interactive entertainment, requires a high degree of automation and precise configuration based on user profiles. Conventional systems struggle to meet these requirements and are unable to provide a fully comfortable living environment for users.

[1671] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1672] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, means for acquiring device status and setting information at system startup, means for reading a user profile and setting the settings based on the individual information, and means for monitoring the user's device usage and optimizing energy efficiency to minimize energy consumption at all times. This makes it possible to manage and improve the user's comfort, health, energy efficiency, and entertainment experience in an integrated manner.

[1673] A "behavioral pattern" refers to the actions or choices a user makes repeatedly in a particular environment or situation.

[1674] "Preferences" refer to the preferences or tendencies of a user that are particularly favored in a particular setting or environment.

[1675] "Lighting" refers to artificial light sources in a user's living space.

[1676] "Temperature" refers to the indoor temperature in the user's living space.

[1677] "Music" refers to audio content that is played in the user's living space.

[1678] "Video" refers to video and image content that is visually displayed in the user's living space.

[1679] "Outside temperature" refers to the temperature of the air outside the living space.

[1680] "Predictive adjustment" means predicting future changes based on past data and current conditions, and making appropriate settings in advance.

[1681] "Reaction" refers to the behavior or response a user exhibits in response to a particular stimulus or situation.

[1682] "Selection" refers to a user selecting a specific item or setting from multiple options.

[1683] "Interactive entertainment" refers to entertainment content that changes dynamically in response to user input and actions.

[1684] "Boot up" refers to getting a system or device up and running and ready for use.

[1685] "Device" refers to electronic devices and machines used in the user's living space.

[1686] "Configuration information" refers to the operating conditions and parameters of a device or system.

[1687] A "user profile" refers to a data set that includes a user's personal information, preferences, behavioral patterns, etc.

[1688] "Individual information" refers to information or data that is unique to a particular user.

[1689] "Monitoring" refers to the continuous watching and recording of the operating status of a particular device or system.

[1690] "Energy efficiency" refers to the efficiency with which a particular device or system uses the energy it consumes.

[1691] The present invention relates to a system that makes a user's living space more interactive and comfortable. This system is mainly composed of a server, a terminal, and various devices (lighting, temperature control devices, smart speakers, etc.). An embodiment of the present invention will now be described in detail.

[1692] System configuration

[1693] The system includes the following main elements:

[1694] 1. Server: A central control unit that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[1695] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[1696] 3. Various devices: Devices for adjusting the physical environment, including "integrated devices" and "sensor clusters."

[1697] System Operation

[1698] Preparing the interface

[1699] When the system starts up, the server connects to the "integrated devices" and obtains the status and setting information of each device. This determines the initial state of the entire system. The device (smartphone or tablet) connects to the server via an application, and the user profile is loaded. This prepares the system to perform settings based on the user's individual information.

[1700] Example: When the system starts up, it receives information about the current room temperature and lighting settings from the integrated device. When you start the app on your smartphone and log in, the server retrieves the user's past activity log and settings.

[1701] Providing a personalized experience

[1702] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes the user profile and determines appropriate settings based on the results, which are then sent to the terminal and each device.

[1703] Example: When a user returns home, the living room lights automatically change to a warmer color and the settings are changed to play their favorite music.

[1704] Predictive Comfort Management

[1705] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[1706] Example: If the outside temperature is predicted to be high, the air conditioner will automatically turn on and adjust the set temperature appropriately.

[1707] Device usage monitoring and energy management

[1708] The server monitors the usage of each device and optimizes energy consumption, issuing instructions to turn off unnecessary devices and functions.

[1709] Example: Lights turn off automatically after a certain time has passed since the last time the user operated them.

[1710] Health data analysis and response

[1711] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and sets optimal settings.

[1712] Example: When the user goes to sleep, the lights change to warm colors to automatically create a comfortable sleeping environment.

[1713] Executing interactive functions

[1714] The server analyzes the camera data, recognizes the user's movements, and sends interactive game content in real time to the "integrated device," which projects it onto the walls and ceiling.

[1715] Example: Dynamic graphics are projected onto the wall as a user plays an exercise game with their family.

[1716] Preferences based on video content

[1717] The server obtains data on the video content being played on the terminal, determines the optimal environment settings, and issues instructions to each device.

[1718] Example: When you play a yoga video, the lighting changes to a softer, more relaxing color and the room temperature adjusts appropriately.

[1719] Optimizing the remote work environment

[1720] The server analyzes the user's remote conference schedule and provides optimal environment settings, while the integrated device activates the projector and adjusts the lighting and sound for the remote conference.

[1721] Example: The projector automatically starts 15 minutes before the start of a meeting, and the brightness and sound settings are adjusted to suit the meeting.

[1722] With these functions, users can enjoy a comfortable and efficient living environment. The system adjusts the environment interactively and automatically to provide an optimal environment according to the user's needs.

[1723] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1724] Step 1: Boot up the system and connect devices

[1725] When the system starts up, the server connects to the integrated devices and obtains the status and setting information of each device. This determines the initial state of the entire system. The input is the system startup signal and the response from the integrated devices, and the output is the current status and setting information of each device.

[1726] How it works: When the system starts up, the server sends a signal to the integrated device and receives information about the current room temperature and lighting settings.

[1727] Step 2: Loading the User Profile

[1728] The device (smartphone or tablet) connects to the server via an application and reads the user profile. The input is the device's login information, and the output is profile information that includes the user's past behavioral data and preferences.

[1729] Specific operation: When a user launches the app on their smartphone and logs in, the server retrieves the user's past activity logs and settings.

[1730] Step 3: Apply personalized settings

[1731] The server analyzes the user profile and automatically adjusts lighting, temperature, music, video, etc. based on past behavioral patterns and preferences. The input is the user profile and current environmental data, and the output is automatically adjusted setting instructions.

[1732] Specific operation: When the user returns home, the living room lights will automatically change to a warm color and the settings will change to play their favorite music.

[1733] Step 4: Real-time environmental adjustment

[1734] The server analyzes external weather data and room usage patterns in real time and predictively adjusts the room temperature accordingly. The input is weather data and usage pattern data, and the output is room temperature adjustment instructions.

[1735] Specific operation: If the outside temperature is predicted to be high, the air conditioner will automatically turn on and adjust the set temperature appropriately.

[1736] Step 5: Monitor device usage and manage energy

[1737] The server monitors the usage of each device and optimizes energy consumption. The input is the usage data of each device, and the output is instructions to turn off unnecessary devices and functions.

[1738] Specific operation: The light will automatically turn off after a certain period of time has passed since the user last operated the light.

[1739] Step 6: Analyze and respond to health data

[1740] The device collects the user's sleep and activity data and sends it to a server. The input is the user's health data, and the output is the optimal environmental settings.

[1741] Specific operation: Before the user goes to sleep, the lighting changes to a warm color, automatically creating a comfortable sleeping environment.

[1742] Step 7: Execute Interactive Features

[1743] The server analyzes the camera data and recognizes the user's movements. The input is the camera image data, and the output is instructions for interactive game content.

[1744] Specific Action: Dynamic graphics are projected onto the wall as the user plays an exercise game with their family.

[1745] Step 8: Setting preferences based on video content

[1746] The server obtains the data of the video content being played on the device and determines the optimal environment settings. The input is the video content data, and the output is instructions for the environment settings.

[1747] What it does: When you play a yoga video, the lighting changes to a soft, relaxing color and the room temperature is adjusted appropriately.

[1748] Step 9: Optimize your remote work environment

[1749] The server analyzes the user's remote conference schedule and provides instructions for optimal environment settings. The input is the remote conference schedule data, and the output is instructions for environment settings suitable for the conference.

[1750] Specific operation: The projector automatically starts up 15 minutes before the start of the meeting, and the brightness and sound settings are adjusted appropriately for the meeting.

[1751] (Application example 1)

[1752] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1753] While conventional systems were able to make users' living spaces interactive, they still had challenges in personalizing the customer experience in physical stores and providing individual product information and navigation using augmented reality technology. Furthermore, it was difficult to automatically adjust the in-store environment based on the customer's profile, making it impossible to provide a comfortable shopping experience with appropriate environmental settings.

[1754] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1755] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, means for personalizing the in-store environment to suit the customer's preferences based on the store's customer profile, means for providing product information to customers using augmented reality technology, providing detailed information on related products and navigating the store, and means for setting the optimal environment based on the analysis of the customer's profile. This makes it possible to improve the customer experience in physical stores and provide the optimal shopping environment for each individual customer.

[1756] "User behavior patterns and preferences" refers to the past repeated behavior and personal preference settings of a particular user.

[1757] "Means for automatically adjusting lighting, temperature, music, and video" refers to a mechanism that automatically changes or sets related environmental elements (lighting, temperature, music, and video) based on the user's settings and behavioral patterns.

[1758] "Means for analyzing behavioral patterns and outside temperatures and predictively adjusting the room temperature to an appropriate level" refers to a function that analyzes user behavioral data and external weather information, predicts the ideal room temperature, and adjusts it in advance.

[1759] "Means for providing interactive entertainment in response to user responses and selections" refers to a function that provides entertainment that changes dynamically based on the user's real-time responses and selections.

[1760] "Means to personalize the in-store environment to suit customer preferences based on the store's customer profile" refers to a system that adjusts the in-store environment, including lighting, temperature, and music, based on the profile information of customers who visit the store.

[1761] "Means for providing customers with product information, detailed guidance on related products, and in-store navigation using augmented reality technology" refers to a method or device that uses augmented reality (AR) technology to display product information in the customer's field of vision, and supports guidance on related products and in-store navigation.

[1762] "Means for optimal environmental settings based on customer profile analysis" refers to a function that analyzes customer profile data and automatically adjusts ideal environmental settings (lighting, temperature, music, etc.) based on that data.

[1763] System configuration

[1764] A system for implementing this invention includes the following main elements:

[1765] 1. Server: This is the central control device that analyzes user behavior patterns and preferences and manages and adjusts various settings.

[1766] 2. Terminal: A device operated by the user (smartphone, tablet, smart glasses) that communicates with the server and sends and receives user profile and configuration information.

[1767] 3. Devices: Lighting, temperature control, music playback, augmented reality devices, and other devices that adjust the physical environment and enhance the user's experience.

[1768] System Operation

[1769] Preparing the interface

[1770] The server connects to each device at system startup to obtain status and configuration information. The device connects to the server via an application, and the user profile is loaded. The system is then ready to configure itself based on the user's individual information.

[1771] Providing a personalized experience

[1772] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. It analyzes the user profile and determines the appropriate settings based on the results, sending them to the terminal and each device.

[1773] Examples:

[1774] If a customer prefers dim lighting when watching a movie, the server can automatically dim the lighting when the movie is playing.

[1775] Predictive Comfort Management

[1776] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[1777] Examples:

[1778] If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[1779] Interactive Entertainment

[1780] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[1781] Examples:

[1782] During the climax of a horror movie, the lighting is automatically set to red and the sound effects are emphasized.

[1783] Personalized environment based on store customer profiles

[1784] The server analyzes the profiles of customers who visit the store and personalizes the in-store environment to suit their preferences, allowing customers to enjoy the optimal shopping environment.

[1785] Examples:

[1786] By playing music that customers like and adjusting lighting and temperature appropriately, we provide a comfortable shopping experience.

[1787] Augmented reality technology for product information and navigation

[1788] The device uses augmented reality technology to display product information to the customer's visual field, providing detailed information on related products and navigation.

[1789] Examples:

[1790] As customers walk through the store using the smart glasses, a guide displayed on the glasses navigates them to the products they want to purchase and displays product features and pricing information.

[1791] Prompt Sentence Examples

[1792] "Automatically adjust the in-store environment (lighting, music, temperature) based on the user's profile. Also, if a customer is wearing smart glasses, show them detailed information about related products and help them navigate to the products they are looking for."

[1793] In this way, the system of the present invention can interactively and automatically adjust environmental settings and entertainment based on user behavior and preferences to enhance the in-store customer experience.

[1794] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1795] Step 1:

[1796] The server connects to each device at system startup and obtains status and configuration information.

[1797] Input: Identification and initial configuration data for each device.

[1798] Output: Current status and configuration information for each device.

[1799] The server collects the setting information sent by each device and adjusts the initial settings based on this.

[1800] Step 2:

[1801] The device connects to the server via the application and the user profile is loaded.

[1802] Input: User credentials and profile data.

[1803] Output: The user's individual profile.

[1804] The terminal launches the application and connects to the server to request and retrieve the user's profile.

[1805] Step 3:

[1806] The server automatically adjusts lighting, temperature, music, video, etc. based on the user's behavioral patterns and preferences.

[1807] Input: User behavioral pattern data and preference settings.

[1808] Output: Adjustment instructions for each device.

[1809] The server analyzes the user's past behavioral patterns and preferences, and based on this calculates appropriate lighting, temperature, music, and video settings and sends them to the relevant devices.

[1810] Step 4:

[1811] The server analyzes external weather data and room usage patterns in real time and predictively adjusts the room temperature to an appropriate level.

[1812] Input: External weather data and room usage pattern data.

[1813] Output: Room temperature control command.

[1814] The server acquires external weather data, compares it with the user's room usage pattern, predicts the room temperature adjustment value, and sends it to the temperature adjustment device.

[1815] Step 5:

[1816] As entertainment content plays on the device, the server analyzes the user's responses and selections in real time and dynamically changes lighting and sound settings.

[1817] Input: entertainment content data and user reaction data.

[1818] Output: Instructions to change lighting and sound settings.

[1819] The device plays entertainment, and the server collects user reactions in real time from sensor data and adjusts lighting and sound accordingly.

[1820] Step 6:

[1821] The server analyzes the profiles of customers who visit the store and personalizes the in-store environment to suit their preferences.

[1822] Input: Customer profile data.

[1823] Output: Adjustment instructions to each device in the store.

[1824] The server analyzes the profile information of customers visiting the store and issues instructions to each device in the store to adjust lighting, music, temperature, etc. based on the results.

[1825] Step 7:

[1826] The device uses augmented reality technology to display product information to the customer's visual field, providing detailed information on related products and navigation.

[1827] Input: Customer navigation request and product information data.

[1828] Output: Augmented reality display data.

[1829] The terminal (such as smart glasses) retrieves product information from the server based on the customer's request, displays it using augmented reality technology, and guides the customer to the location of the desired product.

[1830] This will improve the customer experience in physical stores and provide an optimal shopping environment for each individual customer.

[1831] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1832] The system of the present invention enables more advanced personalization by combining means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for predictively adjusting the room temperature by analyzing the outside temperature, and means for providing interactive entertainment, as well as an emotion engine that recognizes the user's emotions.

[1833] System configuration

[1834] The system includes the following main elements:

[1835] 1. Server: The central control unit that analyzes user behavior patterns, preferences, and emotions, and manages and adjusts various settings.

[1836] 2. Terminal: A device operated by the user (smartphone, tablet, etc.) that communicates with the server and sends and receives user profiles and setting information.

[1837] 3. Emotion engine: An analysis device that recognizes emotions from the user's facial expressions, voice, text data, etc. and reflects them in the settings of the entire system.

[1838] 4. Lighting, thermostats, smart speakers, etc.: Devices for adjusting the physical environment, including integrated devices such as the "Smart Ceiling Nexus."

[1839] System Operation

[1840] Preparing the interface

[1841] When the system starts up, the server connects to "SmartCeiling Nexus" to obtain device status and configuration information. The device (smartphone or tablet) connects to the server via the application, and the user profile is loaded. This prepares the system to configure settings based on the user's individual information.

[1842] Providing a personalized experience

[1843] The server determines personalized settings for lighting, temperature, music, video, etc. based on the user's past behavioral patterns and preferences. To do this, it analyzes data from the user profile and emotion engine, determines appropriate settings based on the results, and sends them to the device and "SmartCeiling Nexus."

[1844] Example: If the emotion engine detects that the user is feeling stressed, the server will set soft lighting and calming music to enhance relaxation.

[1845] Predictive Comfort Management

[1846] The server analyzes external weather data and room usage patterns in real time to predictively adjust the room temperature to an appropriate level, ensuring users stay in a comfortable environment.

[1847] Example: If the outside temperature is predicted to be high, the server will turn on the air conditioning in advance to keep the room comfortable.

[1848] Interactive Entertainment

[1849] As entertainment content is played on the terminal, the server analyzes the user's responses, preferences and emotions in real time and dynamically changes lighting and sound settings.

[1850] Example: During the climax of a horror movie, if the user is recognized as feeling scared, the lighting will be automatically set to red and the sound effects will be enhanced.

[1851] Optimizing energy efficiency

[1852] The server monitors the usage of each device and optimizes energy consumption by issuing instructions to the SmartCeiling Nexus to turn off unnecessary devices and functions, minimizing energy consumption.

[1853] Example: Turning off the lights automatically after a certain time has passed after the user leaves the room.

[1854] Health and Wellness Support

[1855] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and configures it to provide an appropriate environment.

[1856] Example: Before bedtime, if the emotion engine detects that the user is in a state of relaxation, it will change the lighting to a warmer color and keep the temperature optimal.

[1857] Interactive wall / ceiling games

[1858] The server analyzes the camera data and works in conjunction with the emotion engine to recognize the user's movements and emotions. Based on this, interactive game content is sent in real time to the "SmartCeiling Nexus" and projected onto the walls and ceiling.

[1859] Example: When a user plays a jogging game, if the emotion engine recognizes that the user is having fun, it will randomly add fun elements (light and sound effects).

[1860] Environment-based visual experiences

[1861] The server obtains data on the video content being played on the device, determines the optimal environmental settings (lighting, temperature, music), and issues instructions to "SmartCeiling Nexus."

[1862] Example: When playing a yoga video, if the emotion engine detects a strong sense of relaxation, it will soften the lighting and set the temperature optimally.

[1863] Optimizing remote work

[1864] The server analyzes the user's remote meeting schedule and provides the optimal environment settings. "SmartCeiling Nexus" starts the projector and adjusts the lighting and sound for the remote meeting.

[1865] Example: Before a remote meeting begins, if the emotion engine detects stress, it will turn the lights up brighter and play calming music in the background.

[1866] These functions allow users to enjoy a comfortable and efficient living environment. The system of the present invention adjusts the environment interactively and automatically, providing an optimal environment according to the user's needs and emotions.

[1867] The processing flow will be explained below.

[1868] Step 1:

[1869] The server connects to the SmartCeiling Nexus when the system starts up and obtains device status and configuration information.

[1870] Step 2:

[1871] The terminal starts a dedicated application and transmits user authentication information to the server.

[1872] Step 3:

[1873] After verifying the authentication information, the server reads the user profile and analyzes data on past behavioral patterns and preferences.

[1874] Step 4:

[1875] The emotion engine analyzes the user's facial expressions and voice and generates emotion data.

[1876] Step 5:

[1877] The server determines personalized settings such as lighting, temperature, music, and video based on the analysis results and emotional data.

[1878] Step 6:

[1879] The setting information determined by the server is sent to the terminal and "SmartCeiling Nexus."

[1880] Step 7:

[1881] "SmartCeiling Nexus" configures the physical environment, such as adjusting the color temperature of lighting, based on the setting information it receives.

[1882] Step 8:

[1883] The server collects real-time external weather data and room usage patterns to predictively determine a comfortable room temperature.

[1884] Step 9:

[1885] Based on instructions from the server, "SmartCeiling Nexus" automatically adjusts the heating and cooling settings.

[1886] Step 10:

[1887] When entertainment content is played on the terminal, the playback information is sent to the server.

[1888] Step 11:

[1889] The emotion engine analyzes users' reactions in real time while watching entertainment and generates emotion data.

[1890] Step 12:

[1891] The server sends instructions to the SmartCeiling Nexus to dynamically change lighting and sound settings based on the type of content and emotional data.

[1892] Step 13:

[1893] The SmartCeiling Nexus adjusts lighting and sound to provide an interactive entertainment experience.

[1894] Step 14:

[1895] The server monitors the energy usage of each device and issues instructions to turn off unnecessary devices.

[1896] Step 15:

[1897] "SmartCeiling Nexus" follows instructions from the server and automatically changes settings to optimize energy efficiency (e.g., turning off lights that have not been used for a long time).

[1898] Step 16:

[1899] The user's sleep and activity data collected on the device is sent to a server.

[1900] Step 17:

[1901] The emotion engine analyzes the user's emotions before sleep and checks their state of relaxation.

[1902] Step 18:

[1903] The server analyzes this data and emotional data to determine settings (e.g., lighting, temperature) to provide an optimal sleeping environment.

[1904] Step 19:

[1905] The server sends instructions to the SmartCeiling Nexus, which adjusts the color temperature of the lighting and room temperature to create an optimal sleeping environment.

[1906] Step 20:

[1907] The server detects the start of an interactive game and analyzes real-time video data from the camera.

[1908] Step 21:

[1909] The emotion engine analyzes the user's emotions during the game in real time and generates emotion data.

[1910] Step 22:

[1911] The server transmits the game content to the "SmartCeiling Nexus" in real time, where it is projected onto the walls and ceiling.

[1912] Step 23:

[1913] Data of the video content to be played on the terminal is transmitted to the server.

[1914] Step 24:

[1915] The emotion engine analyzes the user's emotions while watching the video and confirms that they are relaxed.

[1916] Step 25:

[1917] The server determines the optimal environmental settings for the video and sends that setting information to "SmartCeiling Nexus."

[1918] Step 26:

[1919] SmartCeiling Nexus adjusts lighting, temperature and music based on user instructions to provide an environment for relaxation and improved concentration.

[1920] Step 27:

[1921] The server analyzes the user's calendar information and checks for remote meeting schedules.

[1922] Step 28:

[1923] The emotion engine analyzes the user's emotions before a remote meeting and confirms that they are feeling stressed.

[1924] Step 29:

[1925] The server determines the optimal environment settings for the remote meeting and sends instructions to the SmartCeiling Nexus.

[1926] Step 30:

[1927] SmartCeiling Nexus activates the projector and adjusts lighting and sound for remote meetings.

[1928] Each of these processing steps taken allows the system to provide an interactive and comfortable living space for the user.

[1929] Example 2

[1930] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1931] Conventional environmental control systems have difficulty in fine-tuning the environment based on the user's behavioral patterns, preferences, and emotions, and are therefore inadequate in optimizing the environment to suit the user's condition in real time. Furthermore, they do not take into account efficient energy use, and are unable to provide interactive entertainment that reflects the user's health and emotions.

[1932] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for analyzing the behavioral patterns and the outside temperature and predictively adjusting the room temperature to an appropriate level, means for recognizing the user's emotions and providing personalized settings according to the emotions, means for providing interactive entertainment according to the user's reactions and selections, and means for monitoring energy consumption and optimizing energy efficiency by turning off unnecessary devices. This makes it possible to optimize the environment based on the user's real-time state, improve energy efficiency, and provide interactive entertainment according to the user's emotions and health state.

[1933] "User behavior patterns" refer to the behavioral tendencies and routines that a user takes in their daily life, such as the time they return home, the time they go to bed, and the devices they use.

[1934] "Preferences" refer to personal tastes and preferences that a user has for a particular environmental setting, such as a favorite genre of music or a preferred lighting brightness.

[1935] "Lighting" refers to an artificial light source used to brighten a room or space, and whose brightness, color, flashing pattern, etc. can be adjusted.

[1936] "Temperature" refers to the temperature of a room or space, and can be adjusted using air conditioning, heating, cooling, etc.

[1937] "Music" is a combination of sounds reproduced from a sound source, and is selected and reproduced based on the user's preferences.

[1938] "Video" refers to visual content that is presented to users through devices such as televisions, projectors, and monitors.

[1939] "Outside temperature" refers to the temperature of the air outside, and is obtained using weather forecasts and real-time weather data.

[1940] "Emotion" refers to the user's state of mind, such as feelings of joy, sadness, anger, surprise, etc.

[1941] "Emotion recognition" means using sensors and analysis engines to identify emotions from a user's facial expressions, tone of voice, text input, etc.

[1942] "Personalized settings" refers to environmental settings that match a user's specific preferences, behavioral patterns, and emotions, and includes lighting, music, temperature, and video settings that are optimized for each individual user.

[1943] "Real-time monitoring" refers to monitoring that occurs at the present moment, and involves the immediate acquisition and analysis of data from sensors and devices.

[1944] "Optimizing energy efficiency" means minimizing energy consumption and using resources in a sustainable manner.

[1945] "Interactive entertainment" refers to entertainment content that changes depending on the user's reactions and choices, allowing for interactive enjoyment.

[1946] The system of the present invention automatically adjusts environmental settings based on a user's behavioral patterns and preferences, and further provides personalized settings by recognizing the user's emotions in real time using an emotion engine. The system includes the following main components:

[1947] 1. Server:

[1948] The server is the central control device of the system, analyzing user behavior patterns and emotional data to optimize environmental settings. The server is connected to an integrated device called "SmartCeiling Nexus" and acquires device status and setting information. The server dynamically changes environmental settings such as lighting, temperature, music, and video using user profiles and real-time sensor data.

[1949] 2. Terminal:

[1950] The terminal (smartphone or tablet) is the device operated by the user and is responsible for communicating with the server. The user logs in to the system using the terminal and sends their profile and settings information to the server, which allows the server to understand the user's preferences and behavioral patterns.

[1951] 3. Emotion Engine:

[1952] The emotion engine is a device that analyzes the user's facial expressions, voice, text data, etc. to recognize emotions. The emotion engine uses sensors such as cameras and microphones to collect real-time data and analyzes the user's current emotional state based on that data. The analysis results are sent to a server and used to optimize environment settings.

[1953] 4. Environmental control device:

[1954] Environmental control devices such as lighting, air conditioning, and smart speakers are integrated into the "SmartCeiling Nexus," which adjusts the physical environment based on instructions from the server.

[1955] To illustrate, consider the following scenario:

[1956] Scenario 1: Adjusting your environment when you get home

[1957] When a user returns home from work, the device connects to the home Wi-Fi and sends a notification to the server. The server reads the user's profile and analyzes the data obtained from the emotion engine. For example, if the user is feeling stressed, the server instructs the SmartCeiling Nexus to set soft lighting and relaxing music.

[1958] Example prompt sentence:

[1959] "Users are coming home and feeling stressed, so set up soft lighting and calming music to promote relaxation."

[1960] Scenario 2: Entertainment optimization

[1961] When a user watches a horror movie, the device sends playback information to the server. The server uses data from the emotion engine to analyze the user's reaction in real time. For example, if the user is feeling scared, the server instructs SmartCeiling Nexus to dim the lights and enhance the sound effects.

[1962] Example prompt sentence:

[1963] "The user is feeling scared during the climax of a horror movie, so dim the lights and amplify the sound effects."

[1964] Scenario 3: Health Support

[1965] Before the user goes to sleep, activity data is sent from the smartwatch to the server via the device. The server then uses this data and the analysis results of the emotion engine to provide an optimal sleeping environment, for example by turning the lighting to a warm color and setting the room temperature to a comfortable temperature.

[1966] Example prompt sentence:

[1967] "Because users are experiencing relaxation before bed, change the lighting to warmer colors and set the room temperature optimally."

[1968] This system can provide an optimal environment according to the user's real-time condition and improve energy efficiency. In particular, the introduction of an emotion engine enables fine-tuned adjustments to match the user's emotions, which is expected to significantly improve quality of life.

[1969] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1970] Step 1: Initialization and Connection

[1971] When the system starts up, the server first connects to the "environmental adjustment device" (e.g., "SmartCeiling Nexus") to obtain device status and configuration information. This initialization procedure prepares the system.

[1972] Input: Device identification information, network settings

[1973] Data processing: Capture device status information and store it in a database on the server.

[1974] Output: Connection status log, current settings information

[1975] Specific operation: The server detects the "SmartCeiling Nexus" via the network, obtains the status of LED lighting, air conditioning, music playback devices, etc., and creates a log that stores the status.

[1976] Step 2: Loading the User Profile

[1977] The terminal transmits the user's login information to the server, and the server retrieves the user profile and reads the user's preferences and behavioral patterns.

[1978] Input: User authentication information, User ID

[1979] Data processing: Reading and analyzing user profile information from the database

[1980] Output: User's past behavior patterns and preference settings

[1981] Specific operation: When a user logs in with a smartphone, the device sends the user ID and authentication information to the server, which then retrieves relevant information from the database and activates the user profile.

[1982] Step 3: Real-time data collection and emotion recognition

[1983] The emotion engine collects real-time data from sensors such as cameras and microphones, analyzes the user's emotions, and transmits this information to a server.

[1984] Input: Real-time video, audio, and text data

[1985] Data processing: Input data into the emotion analysis model to classify the emotional state

[1986] Output: User's emotional state (e.g., stressed, relaxed, happy)

[1987] Specific operation: When a user enters a room, the emotion engine collects data from the camera and microphone, performs emotion analysis in real time, and sends the results to the server.

[1988] Step 4: Optimize and adjust settings

[1989] The server optimizes specific environmental settings based on the user's past behavioral patterns, real-time data, and emotional state, and transmits the settings information to an "environment adjustment device."

[1990] Input: User profile, real-time emotional data, behavioral patterns, outside temperature data

[1991] Data processing: Comprehensive analysis of input data and calculation of optimal environment settings

[1992] Output: Optimized lighting, temperature, music, and video settings instructions

[1993] Specific operation: If the user is analyzed as feeling stressed, the server instructs the "environmental adjustment device" to set warm lighting and relaxing music.

[1994] Step 5: Sync your entertainment content

[1995] The device plays entertainment content and sends the information to the server, which then adjusts the environment settings based on the emotion data and content information.

[1996] Input: Entertainment content information currently being played, emotional data

[1997] Data processing: Analyzing the type of content and the user's emotional state to determine appropriate environment settings

[1998] Output: Adjusted lighting and sound settings

[1999] Specific operation: When a user watches a horror movie, the device sends playback information to the server, and the server analyzes the emotion engine data and sends instructions to the "environment adjustment device" to dim the lights and emphasize the sound effects.

[2000] Step 6: Monitor and optimize energy efficiency

[2001] The server monitors the usage of each device and optimizes energy efficiency by automatically turning off unnecessary devices.

[2002] Input: Operating status and energy consumption data of each device

[2003] Data processing: Analyze usage patterns and energy consumption to identify unnecessary devices

[2004] Output: Instructions to turn off devices, energy usage reports

[2005] Specific operation: After the user leaves the room, the server sends an instruction to the "environmental control device" to turn off all unnecessary lights and devices after a certain period of time.

[2006] Step 7: Monitor your health

[2007] The device collects the user's sleep and activity data and sends it to a server, which analyzes the data and provides appropriate settings.

[2008] Input: User's sleep data, activity data

[2009] Data processing: Analyze health data and determine optimal environmental settings

[2010] Output: Adjustable lighting and temperature settings

[2011] Specific operation: The device sends activity data obtained by the user from the smartwatch to the server, and the server adjusts the lighting and temperature based on that data.

[2012] Step 8: Controlling the Interactive Game

[2013] The server analyzes the camera data and works in conjunction with an emotion engine to recognize the user's movements and emotions, and provides interactive game content based on this.

[2014] Input: Real-time camera data, emotion data

[2015] Data processing: Analyzing user movements and emotions to adjust the game's appearance and behavior

[2016] Output: Instructions for changing game content, setting visual and sound effects

[2017] How it works: When a user plays an immersive game, the camera and emotion engine collect and analyze data, and based on the results, send instructions to the "environment adjustment device" to adjust effects and difficulty in real time.

[2018] Step 9: Delivering an environment-based visual experience

[2019] The server acquires data on the video content being played on the terminal, determines the optimal environmental settings, and issues instructions to the "environment adjustment device."

[2020] Input: Video content information, real-time emotion data

[2021] Data processing: Analyze content attributes and user sentiment to calculate optimal environment settings

[2022] Output: Adjusted lighting, temperature, music settings

[2023] How it works: When a user plays a yoga video, the server recognizes it and sends instructions to the "environmental control device" to adjust the lighting and temperature to enhance the relaxation effect.

[2024] Step 10: Optimize your remote work environment

[2025] The server analyzes the user's remote meeting schedule and instructs the "environmental adjustment device" on the optimal environment settings based on that.

[2026] Input: Remote meeting schedule information, emotion data

[2027] Data processing: Analyze meeting time and user sentiment to determine appropriate environment settings

[2028] Output: Adjusted lighting, projection settings, and sound settings

[2029] Specific operation: When a user inputs a remote meeting schedule into the device, the server analyzes the information and sends instructions to the "environmental adjustment device" to brighten the lights and play soothing music in the background before the meeting begins.

[2030] These processing steps enable the system to optimize the environment based on the user's real-time condition, providing a comfortable and efficient living environment.

[2031] (Application example 2)

[2032] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2033] While conventional environmental control systems can adjust the environment based on a user's behavioral patterns and the external environment, they lack the ability to dynamically adjust the environment based on the user's emotional state. Furthermore, they lack the ability to link with content distribution services, limiting the personalization of the user experience. The present invention aims to address these issues by integrating content distribution with highly accurate environmental adjustments based on the user's emotions.

[2034] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2035] In this invention, the server includes means for automatically adjusting lighting, temperature, music, and video based on the user's behavioral patterns and preferences, means for analyzing the behavioral patterns and outside temperature and predictively adjusting the room temperature to an appropriate level, means for providing interactive entertainment in response to the user's reactions and selections, and means for recognizing the user's emotions and automatically setting the optimal environment during content delivery, thereby enabling highly personalized environmental adjustment according to the user's emotional state and recommendation and delivery of content based on emotions.

[2036] "User behavior patterns" refers to records of the user's daily activities and behaviors and their tendencies.

[2037] "Preferences" refer to the preferences or tastes that a user has for a particular condition or option.

[2038] An "emotion engine" is a system for recognizing and analyzing emotions from a user's facial expressions, voice, and text data.

[2039] "Lighting" refers to electrical equipment and its functions for adjusting the intensity and color tone of light in a room or space.

[2040] "Temperature" is a parameter for measuring and controlling the hotness or coldness of air and objects within a particular space.

[2041] "Music" is an art form expressed through the combination of sounds, and includes elements such as rhythm, melody, and harmony.

[2042] "Video" means a moving or static visual representation that is displayed on a screen or display.

[2043] "Outside temperature" refers to the temperature of the environment outside the building.

[2044] "Interactive entertainment" refers to entertainment content that users can enjoy through two-way interaction with the system.

[2045] "Content distribution" refers to the provision of media content such as video and music to users over a digital network such as the Internet.

[2046] A "recommendation engine" is a system that suggests appropriate content based on a user's past behavior, preferences, and emotions.

[2047] To implement this invention, it is necessary to build a system that includes ...

Claims

1. means for automatically adjusting lighting, temperature, music, and video based on a user's behavioral patterns and preferences; A means for analyzing the behavioral patterns and the outside temperature and predictively adjusting the room temperature to an appropriate level; A system including means for providing interactive entertainment in response to user responses and selections.

2. Energy efficiency optimization measures to minimize energy consumption; a means for analyzing a user's sleep patterns and activity levels and adjusting lighting and temperature to support wellness; 10. The system of claim 1, further comprising means for recognizing user movements and providing interactive exercise and entertainment.

3. A means to create an environment for relaxation and concentration that works in conjunction with lighting, temperature, and music; 10. The system of claim 1, further comprising means for providing a large screen for video conferencing and supporting remote conferencing.

Images