System

Abstract

To provide a system.SOLUTION: A system includes: means for cooperating with generative artificial intelligence, a home appliance control application, and a home assistant; means for realizing automatic home appliance control; means for realizing monitoring electricity usage; means for collecting data on the electricity usage; means for storing the collected data; means for analyzing the stored data; means for generating a power saving proposal on the basis of an analysis result; and means for presenting the generated power saving proposal to a user. The generative artificial intelligence learns a user's action pattern and automates home appliance control on the basis of a prediction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's smart home environment, the control of home appliances and the management of power consumption depend on individual devices and applications, making unified management and automation difficult.In addition, there are a lack of concrete ways for users to understand their power consumption and save energy. [Means for solving the problem]

[0005] This invention realizes automatic home appliance control and electricity usage monitoring by linking generative AI with a smart home appliance control app and a smart home assistant. Furthermore, it learns user behavior patterns and automates home appliance control based on predictions. It also analyzes electricity usage during specific time periods and for specific home appliances in detail and makes energy-saving suggestions. This enables unified management of the smart home environment and efficient management of power usage. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, generative AI learns user behavior patterns and automates home appliance control based on the results of that learning. For example, if it learns that the user wakes up at 7:00 every morning, it will open the curtains and turn on the coffee maker at that time.

[0029] "Example 2"

[0030] In one embodiment of the present invention, the electricity usage monitoring means analyzes in detail the electricity usage of specific time periods or specific home appliances and makes energy saving suggestions. For example, the means analyzes the time periods when an air conditioner is running or when a microwave oven is used, and presents suggestions to the user on optimizing the usage time and frequency of those home appliances.

[0031] "Example 3"

[0032] Furthermore, as one embodiment of the present invention, a means for inquiring about electricity usage by voice or text is provided. A user can inquire of the smart home assistant, such as "How much electricity have I used this month?", and the smart home assistant will respond to the inquiry based on the electricity usage information obtained from the monitoring means.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The generative AI learns the user's behavioral patterns. For example, it learns that the user wakes up at 7am every morning.

[0036] Step 2: Based on the learned behavioral patterns, the generative AI automates home appliance control, for example, automatically opening the curtains at 7:00 AM, the time when the user wakes up.

[0037] Step 3: Similarly, it also controls the coffee maker to start automatically, so that the user can enjoy a cup of coffee as soon as they wake up.

[0038] "Example 2"

[0039] Step 1: The electricity usage monitoring method analyzes in detail the electricity usage of specific time periods or specific home appliances, for example, analyzing the time periods when the air conditioner is running.

[0040] Step 2: Based on the analysis results, the electricity usage monitoring tool will make energy-saving suggestions to the user, such as optimizing the usage time and frequency of the air conditioner.

[0041] "Example 3"

[0042] Step 1: The user makes a voice or text inquiry to the smart home assistant about their electricity usage, such as "How much electricity did I use this month?"

[0043] Step 2: The smart home assistant responds to the query based on the electricity usage information obtained from the monitoring method, allowing the user to easily understand their electricity usage.

[0044] Example 1

[0045] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] Conventional smart home systems require users to manually operate home appliances, limiting the scope of automation. Furthermore, there is a lack of monitoring of electricity usage and energy-saving suggestions, making it difficult to improve energy efficiency. Furthermore, there is no way to control home appliances based on user behavior patterns, resulting in low user convenience.

[0047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0048] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for monitoring electricity usage, means for inquiring about electricity usage by voice or text, means for collecting user behavioral data, means for analyzing and learning from the behavioral data, means for generating a home appliance control schedule based on the learning results, and means for controlling home appliances based on the home appliance control schedule. This enables automatic home appliance control based on the user's behavioral patterns, thereby improving energy efficiency and user convenience.

[0049] "Generative AI" is an artificial intelligence technology that learns user behavior patterns and automates home appliance control based on the results of that learning.

[0050] A "smart home appliance control app" is application software that allows you to remotely control and manage home appliances using devices such as smartphones and tablets.

[0051] A "smart home assistant" is a digital assistant that uses voice recognition and natural language processing to operate home appliances according to user instructions.

[0052] "Automatic home appliance control" is a function that automatically operates home appliances based on the user's behavioral patterns and schedule.

[0053] "Electricity usage monitoring" is a function that monitors and records the power consumption of home appliances and the entire system in real time.

[0054] The "means for inquiring about electricity usage by voice or text" is an interface that allows users to check current electricity consumption status and past usage history using voice commands or text input.

[0055] "Means for collecting user behavioral data" refers to a function that uses smart home devices and sensors to record user behavior and environmental information.

[0056] "Means for analyzing and learning from behavioral data" refers to algorithms and artificial intelligence technologies that analyze collected user behavioral data and find patterns and regularities.

[0057] The "means for generating a home appliance control schedule based on the learning results" is a function that determines the optimal timing and method for operating home appliances based on the analyzed and learned user behavior patterns.

[0058] The "means for controlling home appliances based on the home appliance control schedule" refers to an interface or control system for actually operating home appliances according to the generated schedule.

[0059] MODE FOR CARRYING OUT THE INVENTION

[0060] This invention relates to a system that uses generative AI to learn user behavior patterns and automates home appliance control based on the learning results. Specific embodiments of this system are described below.

[0061] 1. Program Generation

[0062] The server uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate a program that learns user behavior patterns. This program has the function of collecting and analyzing user behavior data and generating a home appliance control schedule based on the learning results.

[0063] 2. Program Processing

[0064] The server collects user behavior data from smart home devices (e.g., smart speakers, smart lights, smart plugs, etc.), including the time and frequency with which users interact with the devices and data from environmental sensors (e.g., temperature, light intensity, motion detection, etc.).

[0065] The server then inputs the collected behavioral data into a generative AI model, which analyzes and learns the user's behavioral patterns. The AI ​​model then finds patterns in this data and identifies the user's typical behavioral schedule.

[0066] The server then generates a schedule for controlling home appliances based on the learning results. For example, the server might send an instruction to the smart curtain device to open the curtains at 7:00 every morning, and an instruction to the smart light device to turn off the lights at 10:00 every night.

[0067] Finally, the terminal (smart home device) receives instructions from the server and actually controls the home appliances, thereby realizing automation that matches the user's behavioral patterns.

[0068] 3. Specific Examples

[0069] As a concrete example, consider a case where a user has the habit of going to bed at 10 p.m. every night. In this case, the server will set the system to automatically turn off the lights and switch the air conditioner to energy-saving mode at 10 p.m.

[0070] Example prompt sentence:

[0071] Design a system that learns user behavior patterns and automates home appliance control. For example, if the user wakes up at 7:00 every morning, design the system to open the curtains and turn on the coffee maker at that time.

[0072] In this way, the server utilizes generative AI models to realize automated home appliance control to make users' lives more comfortable.

[0073] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0074] Program processing flow

[0075] Step 1: Collect user behavior data

[0076] The server collects user behavior data from smart home devices (e.g., smart speakers, smart lights, smart plugs, etc.), such as the time and frequency at which users operate the devices, and data from environmental sensors (e.g., temperature, illuminance, motion detection, etc.).

[0077] Input: Operation logs from smart home devices, environmental sensor data

[0078] Data processing: Organize and save operation logs and sensor data as time-series data

[0079] Output: Organized behavioral data

[0080] Specific behavior:

[0081] The server records the time when the user says "good morning" from the smart speaker.

[0082] The server records the on / off times of the smart lights.

[0083] Step 2: Analyze and learn from behavioral data

[0084] The server inputs the collected behavioral data into a generative AI model (e.g., GPT-4 (registered trademark)) to analyze and learn the user's behavioral patterns. The AI ​​model finds regularities and patterns in this data and identifies the user's typical behavioral schedule.

[0085] Input: Organized behavioral data

[0086] Data Computation: Using generative AI models to analyze behavioral data and learn patterns

[0087] Output: User behavior patterns

[0088] Specific behavior:

[0089] The server learns the pattern of users saying "good morning" every morning at 7am.

[0090] The server learns the pattern that the smart lights turn off at 10pm every night.

[0091] Step 3: Generate a home appliance control schedule

[0092] The server generates a schedule for controlling home appliances based on the learning results, which includes instructions for operating specific devices at specific times.

[0093] Input: User behavior patterns

[0094] Data processing: Determine the optimal timing and method for operating home appliances based on behavioral patterns

[0095] Output: Home appliance control schedule

[0096] Specific behavior:

[0097] The server sends an instruction to the smart curtain device to open the curtains at 7am every morning.

[0098] The server sends an instruction to the smart lights to turn off the lights at 10pm every night.

[0099] Step 4: Implementing appliance control

[0100] The terminals (smart home devices) receive instructions from the server and actually control the home appliances, thereby realizing automation that matches the user's behavioral patterns.

[0101] Input: Home appliance control schedule

[0102] Data calculation: Operate home appliances according to a schedule

[0103] Output: Actual operation of controlled appliances

[0104] Specific behavior:

[0105] The device receives instructions from the server and opens the curtains at 7:00 every morning.

[0106] The terminal receives instructions from the server to turn off the lights at 10 p.m. every night.

[0107] (Application example 1)

[0108] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] Conventional smart home appliance control systems are specialized for controlling home appliances in the home, and do not support automatic control of equipment in commercial facilities such as brick-and-mortar stores. Furthermore, manually controlling in-store equipment places a heavy burden on store staff, hindering efficient store operations. Furthermore, improper control of in-store equipment can result in energy waste and reduced customer satisfaction. To solve these issues, a system that can learn user behavior patterns and automatically control equipment is needed, even in brick-and-mortar stores.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0111] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for learning user behavior patterns and automatically controlling in-store equipment, and means for controlling in-store equipment. This enables automatic control of equipment based on user behavior patterns even in physical stores, reducing the burden on store staff and enabling efficient store management.

[0112] "Generative AI" is a type of artificial intelligence that learns user behavior patterns and automatically controls itself based on the results of that learning.

[0113] A "smart home appliance control app" is an application that runs on mobile devices such as smartphones and tablets and allows remote control of home appliances.

[0114] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to control home appliances and facilities.

[0115] "Automatic home appliance control" is a function that automatically operates home appliances based on user behavior patterns and predictions.

[0116] "Electricity usage monitoring" is a function that monitors and records the electricity consumption of specific time periods or specific home appliances in real time.

[0117] "A means for querying electricity usage via voice or text" refers to an interface that allows users to check their current electricity consumption status using voice commands or text input.

[0118] "Means of learning user behavior patterns and automatically controlling in-store equipment" refers to a function that analyzes the behavior of store staff and customers and automatically operates store equipment such as lighting, music, and air conditioning based on the results.

[0119] "Means for controlling in-store equipment" refers to systems or devices for manually or automatically operating in-store equipment such as lighting, music, and air conditioning.

[0120] A system for implementing this invention includes generative AI, a smart home appliance control app, a smart home assistant, a means for monitoring electricity usage, a means for inquiring about electricity usage via voice or text, a means for learning user behavior patterns and automatically controlling in-store equipment, and a means for controlling in-store equipment.

[0121] System Program

[0122] The system's program is implemented using programming languages ​​such as Python. The generative AI learns user behavior patterns and automatically controls in-store equipment based on the results of that learning. Specifically, it performs the following processes:

[0123] Processing Description

[0124] 1. Generative AI learning phase

[0125] The server collects and analyzes past behavioral data to learn user behavior patterns, including when store staff open the store and when they operate specific equipment, and uses machine learning algorithms to model these patterns.

[0126] 2. Predicting behavioral patterns

[0127] The server predicts future behavior based on learned behavioral patterns. For example, if it learns that store staff open the store at 9:00 a.m. every morning, it will automatically control the store's equipment to match that time.

[0128] 3. Automatic control of equipment

[0129] Based on predicted behavioral patterns, the server automatically controls in-store equipment, such as turning on lights, playing music, and setting the air conditioning to an appropriate temperature.

[0130] 4. Monitoring electricity usage

[0131] The server monitors and records electricity usage during specific time periods and for specific equipment in real time, preventing energy waste and enabling efficient energy management.

[0132] 5. Voice and text inquiries

[0133] Users can check the current power consumption status using voice commands or text input, allowing them to understand power consumption in real time and operate the equipment as needed.

[0134] Hardware and software used

[0135] Hardware: Servers, smartphones, tablets, smart home assistant devices

[0136] Software: Python, machine learning libraries (e.g., TENSORFLOW (registered trademark), scikit-learn (registered trademark)), smart home appliance control app

[0137] Specific examples

[0138] For example, by inputting the following prompt sentence into a generative AI model, it can learn the behavioral patterns of store clerks.

[0139] Learn the behavioral patterns of store staff who open the store at 9am every morning.

[0140] By using this prompt, the generative AI model can learn the behavioral patterns of store staff and automatically control in-store equipment based on the results, thereby reducing the burden on store staff and enabling efficient store operations.

[0141] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0142] Step 1:

[0143] The server collects user behavior data. Specifically, it obtains data such as when store staff open the store and when specific equipment is operated from smart home appliance control apps and smart home assistants. The input is past behavior data, and the output is a list of collected behavior data.

[0144] Step 2:

[0145] The server analyzes the collected behavioral data and learns the user's behavioral patterns. Specifically, it uses a machine learning algorithm to model the behavioral patterns. The input is a list of collected behavioral data, and the output is a learned behavioral pattern model.

[0146] Step 3:

[0147] The server predicts future behavior based on the learned behavioral patterns. Specifically, it uses a behavioral pattern model to predict what behavior will occur at a specific time. The input is the learned behavioral pattern model, and the output is a list of predicted behaviors.

[0148] Step 4:

[0149] The server automatically controls the equipment in the store based on the predicted behavior. Specifically, it performs operations such as turning on the lights, playing music, and setting the air conditioning to an appropriate temperature. The input is a list of predicted behaviors, and the output is the controlled status of the equipment.

[0150] Step 5:

[0151] The server monitors and records the electricity usage of specific time periods and specific equipment in real time. Specifically, it obtains power consumption data from smart home appliance control apps and smart home assistants and stores it in a database. The input is the power consumption data, and the output is the recorded power consumption data.

[0152] Step 6:

[0153] Users can check the current power consumption status using voice commands or text input. Specifically, they query the server through a smart home appliance control app or smart home assistant. The input is a voice command or text input, and the output is information about the current power consumption status.

[0154] Example 2

[0155] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0156] Conventional household electricity usage monitoring systems do not adequately collect and analyze electricity usage data, making it difficult to provide specific energy-saving suggestions to users. They also lack the functionality to learn user behavior patterns and automate home appliance control based on predictions. This has led to the issue of being unable to achieve efficient power management and energy saving.

[0157] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for linking with the generative artificial intelligence, the home appliance control application, and the home assistant, a means for realizing automatic home appliance control, a means for realizing electricity usage monitoring, a means for inquiring about electricity usage by voice or text, a means for collecting electricity usage data, a means for storing the collected data, a means for analyzing the stored data, a means for generating power-saving suggestions based on the analysis results, and a means for presenting the generated power-saving suggestions to the user. This enables detailed monitoring and analysis of electricity usage and provides specific power-saving suggestions to the user. Furthermore, efficient power management and power saving can be achieved by learning user behavior patterns and automating home appliance control based on predictions.

[0158] "Generative AI" is an AI technology that learns user behavior patterns and generates home appliance control and energy-saving suggestions based on predictions.

[0159] A "home appliance control application" is software for controlling home appliances through devices such as smartphones and tablets.

[0160] A "home assistant" is a device or software that interacts with users through voice or text to control home appliances and provide information.

[0161] "Automatic home appliance control" is a function that automatically adjusts the on / off and settings of home appliances based on user behavior patterns and predictions.

[0162] "Electricity usage monitoring" is the process of recording the electricity usage of each household appliance in real time and collecting it as data.

[0163] "A means of inquiring about electricity usage via voice or text" refers to an interface that allows users to check current electricity usage and past usage history through voice commands or text input.

[0164] The "means of collecting electricity usage data" refers to a system that uses smart meters and IoT devices to obtain real-time electricity usage data for each household appliance.

[0165] "Means for storing collected data" refers to a function that stores collected electricity usage data in a database so that it can be analyzed and referenced later.

[0166] "Means for analyzing stored data" refers to software or algorithms for analyzing stored electricity usage data and detecting usage patterns or anomalies.

[0167] The "means for generating energy-saving suggestions based on the analysis results" refers to a generative artificial intelligence that suggests specific energy-saving methods to users based on the results of data analysis.

[0168] The "means for presenting the generated power saving suggestions to the user" is an interface that notifies the user's terminal of the generated power saving suggestions and presents them visually or audibly.

[0169] The present invention is a system for monitoring the amount of electricity used in a home in detail and providing specific energy-saving suggestions to the user. A specific embodiment of this system will be described below.

[0170] The server first collects real-time electricity usage data from each household appliance using smart meters and IoT devices. These devices record the usage time and power consumption of appliances such as air conditioners and microwave ovens. For example, if the air conditioner is running, it records that the power consumption is 1.5 kWh per hour.

[0171] The server then stores the collected electricity usage data in a database, which contains detailed information about each appliance, such as usage time, frequency, and power consumption. For example, the data might be recorded in the format "October 1, 2023, 10:00 - Air conditioner - 1.5kWh."

[0172] The server analyzes the stored data using data analysis software such as Python and R. Specifically, it extracts electricity usage for specific time periods and for specific home appliances and analyzes usage patterns. For example, it confirms that air conditioner usage is concentrated between 9:00 AM and 5:00 PM.

[0173] The server then uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate power-saving suggestions based on the results of the data analysis. The following prompt is input to the generative AI model: "Generate power-saving suggestions for the user based on the following data. Data: Air conditioner usage time: 9:00 AM to 5:00 PM, Usage frequency: 5 times a day. Suggestion: How to optimize air conditioner usage time."

[0174] The generative AI model generates a suggestion such as, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM. We especially recommend refraining from using it in the morning when the outside temperature is relatively low."

[0175] Finally, the server sends the generated energy-saving suggestions to the user's device. The user can check the suggestions via their smartphone or computer. For example, a notification will be displayed saying, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM."

[0176] In this way, the server can monitor and analyze electricity usage in detail and provide specific energy-saving suggestions to users. It can also learn user behavior patterns and automate home appliance control based on predictions, achieving efficient power management and energy savings.

[0177] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0178] Step 1: Data collection

[0179] The server collects real-time electricity usage data from each home appliance via smart meters and IoT devices. The server receives data on the usage time and power consumption of each appliance as input. For example, if the air conditioner is running, it records that the power consumption is 1.5 kWh per hour. The collected electricity usage data is obtained as output.

[0180] Step 2: Save data

[0181] The server stores the collected electricity usage data in a database. As input, the data collected in step 1 is provided. For example, the data is recorded in the format "October 1, 2023 10:00 - Air Conditioner - 1.5kWh". As output, the electricity usage data stored in the database is obtained.

[0182] Step 3: Data analysis

[0183] The server uses data analysis software such as Python or R to analyze the stored data. The electricity usage data stored in the database is provided as input. Specifically, the server extracts electricity usage for specific time periods and for specific home appliances and analyzes usage patterns. For example, it can confirm that air conditioner usage is concentrated between 9:00 AM and 5:00 PM. The analysis results are obtained as output.

[0184] Step 4: Generate power saving suggestions

[0185] The server uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate power-saving suggestions based on the results of the data analysis. The analysis results obtained in step 3 are provided as input. The following prompt sentence is input to the generative AI model: "Please generate power-saving suggestions for the user based on the following data. Data: Air conditioner usage time: 9:00 AM to 5:00 PM, usage frequency: 5 times a day. Suggestion: How to optimize air conditioner usage time." The generated power-saving suggestions are obtained as output.

[0186] Step 5: Present your proposal

[0187] The server sends the generated power-saving suggestions to the user's device. As input, the power-saving suggestions generated in step 4 are provided. The user can check the suggestions through their smartphone or computer. For example, a notification may be displayed saying, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM." As output, the power-saving suggestions presented to the user are obtained.

[0188] (Application example 2)

[0189] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0190] Conventional electricity usage monitoring systems are often only applied to household appliances, making it difficult to optimize electricity usage and propose energy-saving measures in physical stores. In addition, there was a lack of concrete means for store managers to grasp electricity usage in real time and take appropriate energy-saving measures. As a result, the efficiency of electricity consumption in physical stores was not fully achieved.

[0191] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for linking the generative AI with a smart home appliance control app and a smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for monitoring the electricity usage of each home appliance in the store in real time, means for analyzing usage patterns of specific time periods and specific home appliances, means for notifying the store manager of energy-saving suggestions, and means for displaying past energy usage data in graphs to visualize the energy-saving effects. This enables optimization of electricity usage in physical stores and efficient energy-saving measures.

[0192] "Generative AI" is a type of artificial intelligence that generates new information and predictions based on data.

[0193] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[0194] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to comprehensively manage and control home appliances and systems.

[0195] "Automatic home appliance control" is a function that automatically operates and manages home appliances based on user settings and behavioral patterns.

[0196] "Electricity usage monitoring" is a function that measures and records the electricity consumed by home appliances and systems in real time.

[0197] "A means for querying electricity usage via voice or text" is an interface that allows a user to obtain information about electricity usage via voice commands or text input.

[0198] "Means for monitoring the electricity usage of each home appliance in the store in real time" is a function that measures the electricity consumption of home appliances installed in a physical store in real time and collects data.

[0199] "Means for analyzing usage patterns during specific time periods or for specific home appliances" refers to a function that analyzes usage trends during specific time periods or for specific home appliances based on collected electricity consumption data.

[0200] The "means for notifying the store manager of power saving suggestions" is a function that notifies the store manager of methods for reducing power consumption and optimal usage patterns based on the analysis results.

[0201] "A means of displaying past electricity usage data in a graph and visualizing the effects of energy saving" is a function that visually displays power consumption data collected in the past and clearly shows the effects of energy saving.

[0202] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[0203] System Configuration

[0204] This system uses the following hardware and software:

[0205] Hardware:

[0206] Smart meter: Measures the electricity usage of each appliance in the store in real time.

[0207] Server: Collects, stores, and analyzes data.

[0208] Smartphone or tablet: A device used by the store manager to operate the application.

[0209] software:

[0210] Python: A programming language for data analysis and visualization.

[0211] Pandas: A library for data preprocessing and aggregation.

[0212] Matplotlib: A library for generating graphs.

[0213] Smart home appliance control app: An application for remotely controlling home appliances.

[0214] Smart home assistant: Software that manages and controls home appliances and systems in an integrated manner.

[0215] Data processing and calculation

[0216] The server receives electricity usage data collected from smart meters in real time and stores it in a database. Next, it preprocesses the data using Pandas to analyze usage patterns for specific time periods and specific appliances. The analysis results are visualized as graphs using Matplotlib.

[0217] Generative AI learns user behavior patterns from collected data, automates home appliance control based on predictions, and generates energy-saving suggestions that are notified to store managers via a smart home appliance control app.

[0218] Specific examples

[0219] For example, if air conditioner usage peaks at 2 p.m., the generative AI will generate a power-saving suggestion such as, "Air conditioner usage peaks at 2 p.m. Consider shifting usage to the morning." This suggestion is then notified to the store manager via the smart home appliance control app.

[0220] Example prompt sentence:

[0221] "Peak air conditioning usage time is 2 p.m. Consider shifting usage to the morning."

[0222] In this way, it becomes possible to optimize electricity usage in physical stores and take efficient energy-saving measures.

[0223] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0224] Step 1:

[0225] The server receives real-time electricity usage data from the smart meter. The input is the electricity usage data from the smart meter, and the output is the raw data stored in the database in the server. Specifically, the server receives data packets sent from the smart meter and records them in the database.

[0226] Step 2:

[0227] The server uses Pandas to preprocess the received electricity usage data. The input is the raw data saved in step 1, and the output is the preprocessed clean data. Specifically, the server complements missing values ​​in the data and aligns the data based on timestamps.

[0228] Step 3:

[0229] The server analyzes the usage patterns of specific time periods and specific appliances based on the preprocessed data. The input is the clean data obtained in step 2, and the output is the analysis results of the usage patterns. Specifically, the server uses Pandas to group the data and aggregate the usage of each appliance by time period.

[0230] Step 4:

[0231] The server uses generative AI to learn user behavior patterns and automates home appliance control based on predictions. The input is the analysis results of usage patterns obtained in step 3, and the output is the automated home appliance control settings. Specifically, the server uses a machine learning algorithm to model behavior patterns and generate an optimal home appliance control schedule.

[0232] Step 5:

[0233] The server generates power-saving suggestions using generative AI and notifies the store manager through the smart home appliance control app. The input is the automated home appliance control settings obtained in step 4, and the output is a notification of the power-saving suggestions. Specifically, the server generates the power-saving suggestions using the generative AI model and sends them to the app as prompts.

[0234] Step 6:

[0235] The terminal (smartphone or tablet) receives the power-saving suggestion sent from the server and notifies the store manager. The input is the notification of the power-saving suggestion generated in step 5, and the output is a notification to the store manager. Specifically, the terminal displays the power-saving suggestion using the push notification function.

[0236] Step 7:

[0237] The user (store manager) checks the power-saving suggestions through the terminal and adjusts the appliance usage schedule as necessary. The input is the power-saving suggestions received in step 6, and the output is the adjusted appliance usage schedule. Specifically, the user operates the app interface to change the appliance settings.

[0238] Step 8:

[0239] The server displays past electricity usage data in a graph to visualize the power saving effect. The input is the clean data obtained in step 2 and the analysis results of the usage pattern obtained in step 3, and the output is the power saving effect displayed as a graph. Specifically, the server uses Matplotlib to visualize the data and generate the graph.

[0240] Example 3

[0241] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0242] In conventional smart home systems, users had to manually open a dedicated application to check their electricity usage. Furthermore, when using a voice assistant to inquire about electricity usage, it was difficult to obtain accurate data or generate natural responses. Furthermore, there was a lack of technology to accurately understand the user's intent and provide appropriate responses. This resulted in a poor user experience and impaired the convenience of smart home systems.

[0243] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0244] In this invention, the server includes means for converting voice to text, means for analyzing the text and understanding the user's intent, means for acquiring electricity usage data, means for generating a response based on the acquired data, and means for conveying the generated response to the user by voice. This makes it possible to acquire accurate data and generate a natural response when a user inquires about electricity usage by voice or text.

[0245] "Generative AI" is an artificial intelligence technology that analyzes user input and generates answers and predictions in natural language.

[0246] A "smart home appliance control app" is application software that allows you to remotely control home appliances using devices such as smartphones and tablets.

[0247] A "smart home assistant" is a device that uses voice recognition and natural language processing to respond to users' voice commands and control home appliances and systems.

[0248] "Automatic home appliance control" is a function that automatically operates home appliances based on user settings and behavior patterns.

[0249] "Electricity usage monitoring" is a function that monitors and records the amount of electricity consumed within the home in real time.

[0250] A "speech-to-text means" is a technology for converting a user's voice input into textual information.

[0251] "Means for analyzing text and understanding user intent" refers to technology that analyzes text data to understand what the user is looking for.

[0252] "Means for acquiring electricity usage data" refers to a technique for collecting electricity consumption data from the monitoring system.

[0253] "Means for generating answers based on acquired data" refers to technology for analyzing collected data and generating appropriate answers to user inquiries.

[0254] The "means for conveying the generated answer to the user by voice" is a technology for converting the text-format answer into voice and conveying it to the user.

[0255] MODE FOR CARRYING OUT THE INVENTION

[0256] The present invention relates to a smart home system that allows users to inquire about electricity usage by voice or text. Specific embodiments of this system will be described below.

[0257] System configuration

[0258] The system consists of the following major components:

[0259] 1. Smart Home Assistant (Device)

[0260] 2. Server

[0261] 3. Monitoring System

[0262] Hardware and software used

[0263] Speech Recognition Software: Google® Cloud Speech-to-Text API

[0264] Generative AI model: GPT-4(registered trademark) by OpenAI(registered trademark)

[0265] Text-to-speech software: Amazon Polly or Google® Cloud Text-to-Speech

[0266] Monitoring systems: smart meters and energy monitoring systems

[0267] Program processing

[0268] When a user asks the smart home assistant, "How much electricity have I used this month?", the following process takes place:

[0269] 1. Speech-to-text

[0270] The device records the user's voice and sends the voice data to the Google® Cloud Speech-to-Text API, which converts the voice into text and returns it to the device.

[0271] 2. Sending a text

[0272] The terminal sends the converted text to the server. The text data is sent to the server as an HTTP request.

[0273] 3. Text Analysis

[0274] The server analyzes the received text and uses a generative AI model (OpenAI's GPT-4®) to understand the user's intent. The generative AI model generates an appropriate prompt.

[0275] 4. Obtaining electricity usage data

[0276] The server calls the monitoring system's API to retrieve electricity usage data, and may also retrieve past data from a database.

[0277] 5. Answer Generation

[0278] The server uses the electricity usage data to generate answers to user queries, again in natural language using a generative AI model.

[0279] 6. Sending answers and voice synthesis

[0280] The server sends the generated answer to the device, which then verbally communicates the answer to the user. The voice synthesis uses Amazon Polly and Google Cloud Text-to-Speech.

[0281] Specific examples

[0282] As a concrete example, consider a scenario in which a user asks a smart home assistant, "How much electricity have I used this month?" In this case, the following prompt sentence is input to the generative AI model:

[0283] Example prompt sentence:

[0284] A user asks, "How much electricity did I use this month?" Data obtained from a monitoring system shows that the electricity usage this month is 150 kWh. Use this information to generate a natural answer for the user.

[0285] Based on this prompt, the generative AI model generates the following answer:

[0286] "This month's electricity usage is 150 kWh."

[0287] This answer is sent to the terminal, which then uses speech synthesis software to communicate it to the user. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0288] Step 1:

[0289] The user asks the smart home assistant by voice, "How much electricity have I used this month?"

[0290] Input: User's voice

[0291] Output: Audio data

[0292] Specific action: The user speaks to the smart home assistant.

[0293] Step 2:

[0294] The device records the user's voice and sends the voice data to the Google® Cloud Speech-to-Text API, which converts the voice into text and returns it to the device.

[0295] Input: Audio data

[0296] Output: Text data

[0297] Specific operations: The device sends voice data to the Google® Cloud Speech-to-Text API and receives text data.

[0298] Step 3:

[0299] The terminal sends the converted text to the server.

[0300] Input: Text data

[0301] Output: Text data as an HTTP request

[0302] Specific operation: The device sends text data to the server as an HTTP request.

[0303] Step 4:

[0304] The server analyzes the received text and uses a generative AI model (OpenAI's GPT-4®) to understand the user's intent. The generative AI model generates an appropriate prompt.

[0305] Input: Text data

[0306] Output: prompt statement

[0307] Specific operation: The server inputs text data into the generative AI model and generates a prompt sentence.

[0308] Step 5:

[0309] The server calls the monitoring system's API and obtains electricity usage data.

[0310] Input: prompt statement

[0311] Output: Electricity usage data

[0312] Specific operation: The server calls the monitoring system's API and obtains electricity usage data.

[0313] Step 6:

[0314] The server uses the electricity usage data to generate answers to user queries, again in natural language using a generative AI model.

[0315] Input: Electricity usage data

[0316] Output: A natural language response

[0317] How it works: The server inputs electricity usage data into a generative AI model and generates a response in natural language.

[0318] Step 7:

[0319] The server sends the generated answer to the device, which then verbally communicates the answer to the user. The voice synthesis uses Amazon Polly and Google Cloud Text-to-Speech.

[0320] Input: Natural language response

[0321] Output: Audio data

[0322] Specific operations: The server sends the generated answer to the device as an HTTP response. The device inputs the received text into speech synthesis software to generate voice data. The device plays the generated voice to the user.

[0323] (Application example 3)

[0324] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] Conventional smart home systems monitor electricity usage within the home and provide functions that allow users to inquire about it by voice or text, but similar functions are also required in physical stores. In particular, there is a need for a system that monitors energy usage in physical stores in real time, allowing store staff to easily check energy usage and receive energy-saving advice. However, such a system does not currently exist, and energy management is not being carried out efficiently.

[0326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0327] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for monitoring energy usage in the physical store in real time, means for store staff to inquire about energy usage by voice or text using a smartphone, and means for providing energy saving advice based on energy usage data. This enables efficient management of energy usage and power saving even in physical stores.

[0328] "Generative AI" is artificial intelligence that learns user behavior patterns and optimizes home appliances and energy usage based on predictions.

[0329] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[0330] A "smart home assistant" is a device that uses voice recognition and natural language processing to operate home appliances in response to user instructions.

[0331] "Automatic home appliance control" is a function that automatically operates home appliances based on user behavior patterns and predictions.

[0332] "Electricity usage monitoring" is a function that monitors electricity usage in homes and physical stores in real time and collects data.

[0333] "Voice or text electricity usage inquiry means" refers to an interface that allows a user to check electricity usage using voice commands or text input.

[0334] "Means for monitoring energy usage in physical stores in real time" refers to a system for monitoring energy consumption in physical stores in real time and collecting data.

[0335] "A means for store staff to inquire about energy usage by voice or text using a smartphone" is an interface that allows store staff to check energy usage by voice commands or text input using a smartphone.

[0336] The "means for providing energy saving advice based on energy usage data" is a function that analyzes collected energy usage data and provides advice to promote efficient energy usage.

[0337] As an embodiment of the present invention, a management system for energy usage in a physical store is constructed. The system includes generative AI, a smart home appliance control app, a smart home assistant, an electricity usage monitoring means, a means for inquiring about electricity usage by voice or text, a means for monitoring energy usage in the physical store in real time, a means for store staff to inquire about energy usage by voice or text using a smartphone, and a means for providing energy saving advice based on energy usage data.

[0338] System program generation

[0339] The system is built using a Python program, using the speech_recognition library for speech recognition and the pyttsx3 library for speech synthesis, allowing store staff to inquire about energy usage by voice or text using their smartphones.

[0340] Program processing explanation

[0341] The server receives voice input via a microphone and converts the speech to text using the speech_recognition library. It then queries the user for energy usage based on the converted text. It then obtains the corresponding energy usage from mock data and responds via voice using the pyttsx3 library.

[0342] Hardware and software used

[0343] Hardware: Smartphone (microphone, speaker)

[0344] Software: Python, speech_recognition library, pyttsx3 library

[0345] Specific examples

[0346] For example, if a store staff member asks into their smartphone, "How much electricity did you use this month?", the application will respond, "This month's electricity usage is 350 kWh." If they ask, "How much electricity did you use last month?", the application will respond, "Last month's electricity usage was 400 kWh."

[0347] Prompt Sentence Examples

[0348] "How much electricity did you use this month?"

[0349] Please tell me how much electricity you used last month.

[0350] In this way, efficient management of energy usage and power saving can be achieved even in physical stores.

[0351] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0352] Step 1:

[0353] The user launches the smartphone application and asks, "How much electricity have I used this month?"

[0354] Input: User's voice command

[0355] Output: Audio data

[0356] Step 2:

[0357] The terminal uses a microphone to capture the user's voice.

[0358] Input: Audio data

[0359] Output: Captured audio data

[0360] Step 3:

[0361] The device converts the acquired voice data into text using the speech_recognition library.

[0362] Input: Captured audio data

[0363] Output: Text data (e.g., "How much electricity did you use this month?")

[0364] Step 4:

[0365] The terminal analyzes the converted text data and recognizes that it is an inquiry about energy usage.

[0366] Input: Text data

[0367] Output: Analysis results (energy consumption query)

[0368] Step 5:

[0369] The server obtains the corresponding energy usage amount from the mock data based on the analysis results.

[0370] Input: Analysis results

[0371] Output: Energy usage data (e.g. 350 kWh)

[0372] Step 6:

[0373] The server converts the acquired energy usage data into a text response.

[0374] Input: Energy usage data

[0375] Output: Text response (e.g., "This month's electricity usage is 350 kWh.")

[0376] Step 7:

[0377] The device uses the pyttsx3 library to convert the text response into speech and transmits the response to the user through the speaker.

[0378] Input: Text response

[0379] Output: Voice data (e.g., "This month's electricity usage is 350 kWh.")

[0380] Step 8:

[0381] The user listens to the voice response from the terminal and confirms the amount of energy used.

[0382] Input: Audio data

[0383] Output: User understanding (checking energy usage)

[0384] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0385] "Example 1"

[0386] The generative AI works in conjunction with an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, behavioral patterns, etc. For example, if the user is speaking in a sad tone, the emotion engine conveys that information to the generative AI. The generative AI then automates home appliance control based on that information. For example, if the generative AI recognizes that the user is sad, it will perform controls such as dimming the lights in the room.

[0387] "Example 2"

[0388] The emotion engine changes the way it presents the electricity usage monitoring results depending on the user's emotions. For example, if the emotion engine recognizes that the user is angry, it will present the electricity usage monitoring results in a direct and concise manner. On the other hand, if the emotion engine recognizes that the user is relaxed, it will present the electricity usage monitoring results in a detailed and polite manner.

[0389] "Example 3"

[0390] Generative AI automates home appliance control based on information from the emotion engine. For example, if the generative AI recognizes that the user is happy, it can automatically activate a music player and play the user's favorite music. This makes it possible to provide a comfortable living space that responds to the user's emotions.

[0391] The processing flow of each embodiment will be described below.

[0392] "Example 1"

[0393] Step 1: The emotion engine recognizes emotions from the user's tone of voice, facial expressions, behavioral patterns, etc.

[0394] Step 2: The emotion engine communicates this information to the generative AI.

[0395] Step 3: The generative AI uses this information to automate home appliance control. For example, if the AI ​​recognizes that the user is sad, it will dim the lights in the room.

[0396] "Example 2"

[0397] Step 1: The emotion engine recognizes the user's emotion.

[0398] Step 2: The emotion engine changes the way it presents the electricity usage monitoring results according to the user's emotions. For example, if it recognizes that the user is angry, the emotion engine will present the electricity usage monitoring results in a direct and concise manner.

[0399] "Example 3"

[0400] Step 1: The generative AI receives information from the emotion engine.

[0401] Step 2: The generative AI uses this information to automate home appliance control. For example, if the generative AI recognizes that the user is happy, it will automatically activate a music player and play the user's favorite music.

[0402] Example 1

[0403] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] Conventional smart home appliance control systems do not fully consider the user's behavioral patterns and emotions, making it difficult to achieve optimal home appliance control that suits the user's lifestyle. Furthermore, their ability to monitor electricity usage and offer energy-saving suggestions is limited, and improvements to improve user convenience are needed.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0406] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for collecting user behavioral data, means for learning behavioral patterns, means for collecting emotion data, and means for recognizing emotions and linking home appliance control. This enables optimal home appliance control based on the user's behavioral patterns and emotions, and also enables detailed monitoring of electricity usage and energy-saving suggestions.

[0407] "Generative AI" is an artificial intelligence system that learns user behavior patterns and emotions and automates home appliance control based on that.

[0408] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[0409] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to comprehensively manage and control home appliances and systems.

[0410] "Automatic home appliance control" is a function that automatically controls the operation of home appliances based on the user's behavioral patterns and emotions.

[0411] "Electricity usage monitoring" is a function that monitors and records the power consumption of home appliances and systems in real time.

[0412] "Enquire about electricity usage by voice or text" is a function that allows users to check current electricity consumption status and past usage history using voice commands or text input.

[0413] "User behavior data" is data that indicates the behavioral patterns and habits of the user in their daily lives.

[0414] "Learning behavioral patterns" is the process of analyzing collected user behavior data and extracting specific patterns and trends.

[0415] "Emotion data" is data that indicates the emotional state of the user, which can be obtained from the user's tone of voice, facial expression, behavior, etc.

[0416] "Recognizing emotions and coordinating home appliance control" is a function that analyzes the user's emotional state and adjusts the operation of home appliances based on the results.

[0417] MODE FOR CARRYING OUT THE INVENTION

[0418] This invention is a system that realizes automatic home appliance control based on user behavior patterns and emotions by linking generative AI with a smart home appliance control app and a smart home assistant. Specific embodiments of this system are described below.

[0419] 1. Program Generation

[0420] The server learns user behavior patterns and generates a generative AI program that automates home appliance control based on the learning results. This program includes algorithms for collecting and analyzing user behavior data. Specifically, a machine learning model is built using software such as Python and TENSORFLOW (registered trademark).

[0421] 2. Program Processing

[0422] The server uses generative AI to learn the user's behavioral patterns. For example, if the server learns that the user wakes up at 7:00 every morning, it will open the curtains and turn on the coffee maker at that time. This process is performed using the API of the smart home device.

[0423] The generative AI also works in conjunction with the emotion engine. The emotion engine uses OpenCV and NLP (natural language processing) libraries to analyze emotions from the user's tone of voice and facial expressions. For example, if the user speaks in a sad tone, the emotion engine conveys that information to the generative AI. The generative AI then automates home appliance control based on that information. For example, if the generative AI recognizes that the user is sad, it will dim the lights in the room.

[0424] 3. Specific Examples

[0425] As a concrete example, consider a case where a user has the habit of waking up at 7am every morning and drinking coffee. The generative AI learns this pattern and opens the curtains and starts the coffee maker at 7am every morning. Also, if the user says "I'm tired today" in a sad tone of voice, the emotion engine relays this information to the generative AI, which then dims the lights in the room.

[0426] Prompt Sentence Examples

[0427] An example prompt is:

[0428] Explain how the generative AI would control the appliances if the user wakes up at 7am every morning and has the habit of drinking coffee. Also, explain how the generative AI would control the appliances if the user says "I'm tired today" in a sad tone of voice.

[0429] In this way, generative AI automates home appliance control based on user behavior patterns and emotions.

[0430] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0431] Program processing flow

[0432] Step 1: Collect user behavior data

[0433] The server collects user behavior data. Specifically, it acquires user activity data (e.g., heart rate, number of steps, wake-up time, etc.) through sensors in the smartwatch or smartphone.

[0434] Input: Sensor data from smartwatches and smartphones

[0435] Output: User behavior data (e.g., wake-up time, activity level)

[0436] Step 2: Learning behavioral patterns

[0437] The server uses the collected behavioral data to learn user behavior patterns. It uses Python and Tensorflow (registered trademark) to build a machine learning model and analyze user behavior patterns.

[0438] Input: User behavior data

[0439] Output: Learned behavioral patterns (e.g., waking up at 7am every morning)

[0440] Step 3: Automate home appliance control

[0441] The server automatically controls home appliances based on the learned behavioral patterns. It calls the API of the smart home device and operates the appliances according to the user's behavioral patterns. For example, if the server learns that the user wakes up at 7:00 every morning, it will open the curtains and start the coffee maker at that time.

[0442] Input: Learned behavioral patterns

[0443] Output: Home appliance control commands (e.g., open curtains, turn on coffee maker)

[0444] Step 4: Collecting emotion data

[0445] The device collects the user's emotional data, capturing their tone of voice and facial expressions through a smart speaker or camera, and sends the data to a server.

[0446] Input: Audio and video data from smart speakers and cameras

[0447] Output: User emotion data (e.g., tone of voice, facial expression)

[0448] Step 5: Coordination of emotion recognition and home appliance control

[0449] The server uses an emotion engine to recognize the user's emotions. It uses OpenCV and NLP libraries to analyze emotions from the user's tone of voice and facial expressions. Based on the analysis results, the generative AI automates home appliance control. For example, if it recognizes that the user is sad, it will dim the lights in the room.

[0450] Input: User emotion data

[0451] Output: Home appliance control command (e.g. dim the lights)

[0452] Step 6: Monitor your electricity usage

[0453] The server monitors and records the power consumption of home appliances and systems in real time, and provides detailed analysis of the power consumption of specific time periods and specific appliances to suggest ways to save energy.

[0454] Input: Power consumption data from appliances and systems

[0455] Output: Electricity usage report and energy saving suggestions

[0456] In this way, generative AI can automate home appliance control based on the user's behavioral patterns and emotions, while also providing detailed monitoring of electricity usage and energy-saving suggestions.

[0457] (Application example 1)

[0458] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0459] Conventional smart home appliance control systems can automate home appliances based on user behavior patterns, but they have the problem of not being able to adjust the environment based on the user's emotions. Furthermore, in brick-and-mortar stores, there is a need to optimize the store environment based on the emotions and behavior patterns of customers, but there was no system that could do this automatically. This made it difficult to improve customer satisfaction.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0461] In this invention, the server includes means for linking with the generative AI, the smart home appliance control app, and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for linking with an emotion engine that recognizes user emotions and adjusting the store environment, and means for automatically adjusting the lighting, music, and temperature of the store based on the user's behavioral patterns and emotions. This makes it possible to automatically optimize the store environment based on the emotions and behavioral patterns of customers.

[0462] "Generative AI" is artificial intelligence that learns user behavior patterns and emotions and automates the control of home appliances and the environment based on that.

[0463] A "smart home appliance control app" is an application for remotely controlling home appliances through devices such as smartphones and tablets.

[0464] A "smart home assistant" is a device that uses voice recognition and artificial intelligence to control home appliances.

[0465] "Automatic home appliance control" is a function that automatically operates home appliances based on the user's behavioral patterns and emotions.

[0466] "Electricity usage monitoring" is a function that monitors the amount of electricity consumed within a home or store in real time.

[0467] An "emotion engine" is software or hardware that recognizes emotions from a user's tone of voice, facial expressions, and behavioral patterns.

[0468] "Means for adjusting the store environment" refers to a system for controlling environmental elements such as lighting, music, and temperature within the store.

[0469] "Behavioral patterns" refer to the tendencies and habits of a user's daily behavior.

[0470] "Means for automatically adjusting lighting, music, and temperature" refers to a function that automatically adjusts lighting, music, and temperature in the store based on the user's behavioral patterns and emotions.

[0471] The system for implementing this invention includes a generative AI, a smart home appliance control app, a smart home assistant, an emotion engine, and a store environment adjustment means. The specific configuration and operation of the system are described below.

[0472] System configuration

[0473] 1. Generative AI

[0474] Generative AI is an artificial intelligence that learns user behavioral patterns and emotions and automates the control of home appliances and the environment based on that. For example, it is implemented using machine learning libraries such as Python and TENSORFLOW (registered trademark).

[0475] 2. Smart home appliance control app

[0476] The smart home appliance control app is an application for remotely controlling home appliances through devices such as smartphones and tablets. It runs on iOS and Android (registered trademark) platforms.

[0477] 3. Smart Home Assistant

[0478] A smart home assistant is a device that uses voice recognition and artificial intelligence to control home appliances, such as Amazon Echo or GOOGLE® HOME®.

[0479] 4. Emotion Engine

[0480] The emotion engine is software or hardware that recognizes emotions from the user's tone of voice, facial expressions, and behavioral patterns. For example, it uses OpenCV or Microsoft® Azure® emotion recognition APIs.

[0481] 5. Store environment adjustment measures

[0482] Store environment control means are systems for controlling environmental elements such as lighting, music, and temperature within a store, including smart lighting systems (such as Philips® Hue®) and smart thermostats (such as Nest).

[0483] System Operation

[0484] The server uses generative AI to learn the user's behavioral patterns and emotions. The emotion engine analyzes the user's tone of voice and facial expressions in real time and sends the results to the generative AI. The generative AI then uses this data to provide instructions for optimizing the store environment.

[0485] For example, when a customer walks into a store, a smartphone app captures the customer's tone of voice and facial expression. If the customer says in a sad tone, "I'm a little tired today," the app can use that information to dim the store's lights and change the music to something more soothing.

[0486] Specific examples

[0487] As a concrete example, the following prompt sentence is input into the generative AI model:

[0488] Prompt Sentence Examples

[0489] When a customer walks into a store, the smartphone app captures their tone of voice and facial expression. If the customer says in a sad tone, "I'm a little tired today," the app can use that information to dim the store's lights and change the music to something more soothing.

[0490] In this way, it becomes possible to automatically optimize the store environment based on the emotions and behavioral patterns of customers.

[0491] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0492] Step 1:

[0493] The user enters the store.

[0494] Input: User enters the store

[0495] Output: User's store entry information

[0496] Specific operation: The smartphone app detects when a user enters a store and sends the entry information to the server.

[0497] Step 2:

[0498] The device captures the user's tone of voice and facial expressions.

[0499] Input: User's tone of voice and facial expression

[0500] Output: Captured audio and image data

[0501] What it does: It uses the smartphone's microphone and camera to capture the user's tone of voice and facial expressions in real time.

[0502] Step 3:

[0503] The server sends the captured data to the emotion engine.

[0504] Input: Audio data and image data

[0505] Output: Sending data to the emotion engine

[0506] Specific operation: The smartphone app sends the captured audio and image data to the server, which then sends it to the emotion engine.

[0507] Step 4:

[0508] The emotion engine analyzes the user's emotions.

[0509] Input: Audio data and image data

[0510] Output: Parsed emotion data

[0511] Specific operation: The emotion engine analyzes voice and image data and recognizes the user's emotions (e.g., sad, happy, tired).

[0512] Step 5:

[0513] The server sends the analyzed emotion data to the generative AI.

[0514] Input: Parsed emotion data

[0515] Output: Sending data to generative AI

[0516] Specific operation: The emotion engine sends the analyzed emotion data to the server, and the server sends it to the generative AI.

[0517] Step 6:

[0518] Generative AI generates instructions to optimize the store environment based on user behavior patterns and emotions.

[0519] Input: Analyzed emotion data and behavior pattern data

[0520] Output: Store environment optimization instructions

[0521] Specific operation: Generative AI generates instructions to optimize the store's lighting, music, temperature, etc. based on the user's emotional data and past behavioral patterns.

[0522] Step 7:

[0523] The server transmits the generated instructions to the store environment adjustment means.

[0524] Input: Store environment optimization instructions

[0525] Output: Send instructions to store environment adjustment means

[0526] Specific operation: The server sends the instructions received from the generative AI to the store environment adjustment means.

[0527] Step 8:

[0528] The store environment adjusting means adjusts the lighting, music, and temperature of the store based on the instructions.

[0529] Input: Store environment optimization instructions

[0530] Output: Optimized store environment

[0531] Specific operation: Based on instructions from the server, the store environment adjustment means dims the lights, changes the music to a more calming one, and adjusts the temperature.

[0532] Example 2

[0533] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0534] Conventional smart home appliance control systems monitor electricity usage and offer energy-saving suggestions, but they have the problem of being unable to respond flexibly to user emotions. It is also difficult to analyze usage patterns for specific time periods or for each appliance in detail and offer efficient energy-saving suggestions.

[0535] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0536] In this invention, the server includes means for linking with the generative artificial intelligence, the smart home appliance control application, and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for recognizing the user's emotions and presenting the results of the electricity usage monitoring according to the emotions, and means for collecting electricity usage data and analyzing usage status for specific time periods and for each home appliance. This enables flexible responses according to the user's emotions and makes it possible to make efficient energy-saving suggestions by analyzing usage status for specific time periods and for each home appliance in detail.

[0537] "Generative AI" is an AI technology that learns user behavior patterns and automates home appliance control based on predictions.

[0538] A "smart home appliance control application" is software that allows remote control of home appliances using devices such as smartphones and tablets.

[0539] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to manage and control home appliances in an integrated manner.

[0540] "Automatic home appliance control" is a function that automatically operates home appliances based on user behavior patterns and predictions.

[0541] "Electricity usage monitoring" is the process of measuring the electricity usage of home appliances in real time and collecting and analyzing the data.

[0542] "Voice and text electricity usage inquiry means" refers to an interface that allows a user to obtain information about electricity usage using voice commands or text input.

[0543] "Means for recognizing user emotions" refers to technology that uses sensors such as cameras and microphones to analyze the user's facial expressions and tone of voice to determine their emotional state.

[0544] The "means for presenting the monitoring results of electricity usage according to emotions" is a function for displaying the monitoring results of electricity usage in different formats based on the emotional state of the user.

[0545] "Means of collecting electricity usage data" refers to technology that uses smart meters and IoT devices to measure the electricity usage of home appliances and send that data to a server.

[0546] "Means for analyzing usage patterns for specific time periods and for each home appliance" refers to the process of analyzing collected electricity usage data and extracting usage patterns for specific time periods and for each home appliance.

[0547] MODE FOR CARRYING OUT THE INVENTION

[0548] This invention is a system that links generative artificial intelligence with a smart home appliance control application and a smart home assistant, and has the function of presenting the results of monitoring electricity usage according to the user's emotions. Specific embodiments of this system are described below.

[0549] 1. System Overview

[0550] The system consists of three main components: a server, a terminal, and a user. The server collects and analyzes data, the terminal recognizes user emotions, and the user receives the results.

[0551] 2. Hardware and Software Used

[0552] Server: Database management system (e.g., MySQL (registered trademark)), data analysis tool (e.g., Python's Pandas library)

[0553] Device: Camera, microphone, emotion analysis software (e.g., Microsoft® Azure® Emotion API)

[0554] User: Devices such as smartphones and tablets

[0555] 3. Program Processing

[0556] The server collects real-time electricity usage data from each home appliance using smart meters and IoT devices. For example, sensors attached to air conditioners and microwave ovens measure electricity usage and send the data to the server.

[0557] The server stores the collected electricity usage data in a database and analyzes it using the Python Pandas library. Specifically, it extracts usage patterns for specific time periods and for each appliance, and identifies abnormal usage and opportunities for power saving. For example, if an air conditioner is used for more than eight hours a day, the data is flagged.

[0558] To recognize the user's emotions, the device uses a camera and microphone to capture the user's facial expressions and tone of voice. This uses the Microsoft® Azure® Emotion API. For example, when a user speaks in front of the device, the device analyzes their emotions from their tone of voice and facial expressions to determine whether they are angry or relaxed.

[0559] The server presents the results of electricity usage monitoring to the user based on the results of emotion recognition. If the user is angry, the results are presented briefly. For example, a message such as "You are using the air conditioner for too long. Please reduce it by one hour" is displayed. On the other hand, if the user is relaxed, the results are presented in detail. For example, a detailed suggestion such as "You are using the air conditioner for more than eight hours a day. By reducing this by one hour, you can save 500 yen on your monthly electricity bill" is displayed.

[0560] 4. Examples and prompts

[0561] Examples:

[0562] Air conditioner usage:

[0563] The server collects electricity usage data from air conditioners every hour and stores it in a database.

[0564] The server uses Python's Pandas library to flag any usage exceeding eight hours per day.

[0565] The device uses a camera and microphone to recognize emotions when the user is in front of the device.

[0566] If the server recognizes that the user is angry, it will simply say, "You've been using the air conditioner for too long. Please reduce it by one hour."

[0567] Frequency of microwave use:

[0568] The server tallyes the number of times the microwave is used each day and stores the count in a database.

[0569] The server uses Python's Pandas library to flag any usage exceeding five times in a day.

[0570] If the device recognizes that the user is relaxed, it will provide detailed information such as, "You use the microwave a lot. By using multiple devices at once, you can save on electricity bills."

[0571] Example prompt sentence:

[0572] "Analyze air conditioner usage time and provide energy-saving suggestions."

[0573] "When users are relaxed, provide detailed electricity usage monitoring results."

[0574] The system efficiently manages users' electricity usage and responds flexibly to their emotions, helping them live more comfortable and economical lives.

[0575] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0576] Step 1:

[0577] Collecting electricity usage data

[0578] The server collects electricity usage data from each home appliance in real time.

[0579] Input: Electricity usage data sent from smart meters and IoT devices.

[0580] Specific operation: Sensors attached to air conditioners and microwave ovens measure electricity usage and send the data to a server.

[0581] Output: Collected electricity usage data is stored in a database.

[0582] Step 2:

[0583] Data storage and initial processing

[0584] The server stores the collected electricity usage data in a database and performs initial processing.

[0585] Input: Electricity usage data collected in step 1.

[0586] Specific operations: Store data using a database management system (e.g., MySQL (registered trademark)) and clean and format the data using Python's Pandas library.

[0587] Output: Electricity usage data after initial processing.

[0588] Step 3:

[0589] Analyzing the data

[0590] The server analyzes the electricity usage data after the initial processing is completed.

[0591] Input: Electricity usage data after initial processing in step 2.

[0592] How it works: Using Python's Pandas library, it extracts usage patterns for specific time periods and for each appliance, and identifies abnormal usage and opportunities for power saving. For example, if an air conditioner is used for more than eight hours a day, it flags that data.

[0593] Output: Analysis of usage patterns and abnormal usage data.

[0594] Step 4:

[0595] emotion recognition

[0596] The device uses a camera and microphone to capture the user's facial expressions and tone of voice to recognize their emotions.

[0597] Input: The user's facial expression and tone of voice.

[0598] Specific operation: Analyzes the user's emotional state using emotion analysis software (e.g., Microsoft® Azure® Emotion API). For example, when a user speaks to the device, the device analyzes their emotions from their tone of voice and facial expressions to determine whether they are angry or relaxed.

[0599] Output: User's emotional state data.

[0600] Step 5:

[0601] Presentation of results

[0602] The server presents the user with the results of monitoring electricity usage based on the emotion recognition results.

[0603] Input: Analysis results from step 3 and emotional state data from step 4.

[0604] Specific behavior: If the user is angry, the results will be presented succinctly. For example, a message such as "You are using the air conditioner for too long. Please reduce it by one hour" will be displayed. On the other hand, if the user is relaxed, the results will be presented in detail. For example, a detailed suggestion such as "You are using the air conditioner for more than eight hours a day. By reducing this by one hour, you can save 500 yen on your monthly electricity bill" will be displayed.

[0605] Output: Electricity usage monitoring results and energy saving suggestions presented to the user.

[0606] (Application example 2)

[0607] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0608] Conventional electricity usage monitoring systems present information uniformly without considering the user's emotions, which means they are unable to provide appropriate information according to the user's situation and emotions. Furthermore, in physical stores, the lack of detailed analysis of electricity usage and energy-saving suggestions makes efficient energy management difficult.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0610] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for detecting a user's emotions and presenting the results of electricity usage monitoring according to the emotions, means for monitoring electricity usage in a physical store in real time and performing detailed analysis of electricity usage during specific time periods and for specific devices, and means for presenting the results of electricity usage monitoring and making energy saving suggestions according to the emotions of the store manager. This makes it possible to provide appropriate information according to the user's emotions and enable efficient energy management in physical stores.

[0611] "Generative AI" is an artificial intelligence technology that generates new information and predictions based on data.

[0612] A "smart home appliance control app" is an application that allows you to remotely control home appliances using a smartphone or tablet.

[0613] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to comprehensively manage and control home appliances and systems.

[0614] "Automatic home appliance control" is a function that automatically operates home appliances based on user settings and behavior patterns.

[0615] "Electricity usage monitoring" is the process of monitoring and recording the electricity consumption of specific devices or at specific times in real time.

[0616] "Enquire about electricity usage by voice or text" is a function that allows the user to obtain information about electricity usage by voice instructions or text input.

[0617] "Detecting user emotions" refers to a technology that identifies a user's emotional state from their facial expressions, voice, behavior, etc.

[0618] "Presenting electricity usage monitoring results according to emotions" is a function that displays electricity usage information in an appropriate format based on the user's emotional state.

[0619] "Real-time monitoring of electricity usage in physical stores" refers to the process of monitoring electricity consumption in stores in real time and collecting data.

[0620] "Analyzing electricity usage in detail for specific time periods or specific devices" refers to the process of analyzing electricity consumption data in detail for specific time periods or specific devices to evaluate usage patterns and efficiency.

[0621] "Presenting the results of monitoring electricity usage and making energy-saving suggestions based on the store manager's emotions" is a function that displays electricity usage information in an appropriate format and makes suggestions to improve energy efficiency based on the store manager's emotional state.

[0622] A system for implementing this invention includes generative AI, a smart home appliance control app, a smart home assistant, an electricity usage monitoring means, a means for inquiring about electricity usage by voice or text, a means for detecting a user's emotions, a means for presenting the results of monitoring electricity usage according to the emotions, a means for monitoring electricity usage in a physical store in real time, a means for performing detailed analysis of electricity usage during specific time periods or for specific devices, and a means for presenting the results of monitoring electricity usage according to the emotions of the store manager and making energy saving suggestions.

[0623] Explanation of program processing

[0624] The server uses generative AI to learn user behavior patterns and automate home appliance control based on predictions. The smart home appliance control app and smart home assistant receive user voice instructions and text input and acquire electricity usage data. The electricity usage monitoring tool monitors power consumption during specific time periods and for specific devices in real time and collects data.

[0625] The means for detecting user emotions identifies the emotional state of the user from facial expressions, voice, behavior, etc. The means for presenting the monitoring results of electricity usage according to emotions displays the electricity usage information in an appropriate format based on the emotional state of the user. For example, if the user is angry, the information is presented in a concise manner, and if the user is relaxed, the information is presented in a detailed manner.

[0626] The method for monitoring electricity usage in a physical store in real time involves monitoring electricity consumption in the store in real time and collecting data. The method for analyzing electricity usage in detail during specific time periods or for specific devices involves analyzing electricity consumption data in detail during specific time periods or for specific devices to evaluate usage patterns and efficiency.

[0627] The means for presenting the monitoring results of electricity usage and making energy saving suggestions according to the store manager's emotions displays the electricity usage information in an appropriate format and makes suggestions for improving energy efficiency based on the store manager's emotional state.

[0628] Specific examples

[0629] For example, suppose a store manager opens an application on their smartphone. The application monitors the electricity usage of the air conditioner and microwave oven for 24 hours. If the manager's emotion is "angry," the application displays the total usage in a concise manner. If the manager's emotion is "relaxed," the application displays the detailed usage and peak times.

[0630] Example prompts to input to a generative AI model:

[0631] "Please develop an application that monitors electricity usage in physical stores and performs detailed analysis of electricity usage during specific time periods and for specific devices. In addition, please add a function that displays the results of electricity usage monitoring and makes energy-saving suggestions based on the store manager's emotions."

[0632] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0633] Step 1:

[0634] The server uses generative AI to learn user behavior patterns.

[0635] Input: User's past behavior data

[0636] Output: Model of behavioral patterns

[0637] How it works: The server collects data on the user's past behavior and inputs it into a generative AI model. The model learns the user's behavioral patterns and generates a predictive model.

[0638] Step 2:

[0639] The device receives user voice commands and text inputs using a smart home appliance control app and smart home assistant.

[0640] Input: User voice commands and text input

[0641] Output: Home appliance control commands

[0642] Specific operation: The device receives voice instructions or text input from the user, analyzes them, and generates home appliance control commands, which are then sent to the smart home appliances.

[0643] Step 3:

[0644] The server uses an electricity usage monitoring means to monitor the power consumption of a specific time period or a specific device in real time.

[0645] Input: Power consumption data

[0646] Output: Real-time power consumption report

[0647] Specific operation: The server collects power consumption data sent from each device and monitors it in real time. The collected data is compiled into a real-time power consumption report.

[0648] Step 4:

[0649] The terminal uses a means for detecting the user's emotions to identify the user's emotional state from facial expressions, voice, actions, etc.

[0650] Input: User facial, voice, and behavioral data

[0651] Output: Emotional state

[0652] Specific operation: The device collects the user's facial expression, voice, and behavioral data through the camera and microphone, and inputs it into the emotion detection algorithm. The algorithm identifies the user's emotional state and outputs the result.

[0653] Step 5:

[0654] The server uses the means for presenting the monitoring results of electricity usage according to emotions to display the information in an appropriate format based on the emotional state of the user.

[0655] Input: Emotional state, power consumption data

[0656] Output: Emotion-based power consumption report

[0657] Specific operation: The server combines the user's emotional state with the power consumption data to generate a power consumption report according to the user's emotions. For example, if the user is angry, it generates a concise report, and if the user is relaxed, it generates a detailed report.

[0658] Step 6:

[0659] The server monitors the electricity usage of the physical store in real time and performs detailed analysis of electricity usage during specific time periods and for specific devices.

[0660] Input: In-store electricity consumption data

[0661] Output: Detailed power consumption analysis report

[0662] Specific operation: The server collects power consumption data sent from each device in the store and analyzes it in detail, focusing on specific time periods or specific devices. The analysis results are compiled into a detailed power consumption analysis report.

[0663] Step 7:

[0664] The server presents the results of monitoring electricity usage and makes energy saving suggestions according to the store manager's emotions.

[0665] Input: Manager's emotional state, power consumption data

[0666] Output: Energy saving suggestion report according to emotions

[0667] Specific operation: The server combines the manager's emotional state with power consumption data to generate a report of energy-saving suggestions according to the manager's emotions. For example, if the manager is stressed, it will provide brief suggestions, and if the manager is relaxed, it will provide detailed suggestions.

[0668] Example 3

[0669] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0670] In conventional smart home systems, users have limited means to easily check their electricity usage, and home appliances are not controlled according to the user's emotions, making it difficult to provide a comfortable living space. Furthermore, there has been a lack of systems that take into account the user's behavioral patterns and emotions when it comes to monitoring electricity usage and automating home appliance control.

[0671] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0672] In this invention, the server includes means for linking the generative AI with the smart home appliance control application and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for generating instructions for controlling home appliances based on information from the emotion engine, and means for controlling home appliances according to the user's emotions. This allows the user to easily check electricity usage and enjoy a comfortable living space according to their emotions.

[0673] "Generative AI" is artificial intelligence that analyzes users' behavioral patterns and emotions and generates instructions for controlling home appliances based on that.

[0674] A "smart home appliance control application" is software for controlling home appliances in a smart home.

[0675] A "smart home assistant" is a device that receives user instructions via voice or text and operates home appliances and systems accordingly.

[0676] "Automatic home appliance control" is a function that automatically operates home appliances based on the user's behavioral patterns and emotions.

[0677] "Electricity usage monitoring" is a function that monitors and records the amount of electricity consumed within the home in real time.

[0678] The "emotion engine" is a system that analyzes the user's emotions from facial expressions, tone of voice, etc. and provides that information.

[0679] "Instructions for controlling home appliances" are commands for operating home appliances that are generated by generative AI based on the user's emotions and behavioral patterns.

[0680] The "means for controlling home appliances according to the user's emotions" is a function that operates home appliances according to the user's emotions based on information from the emotion engine.

[0681] This invention is a system that allows users to inquire about electricity usage by voice or text, and automates home appliance control based on information from a generative AI emotion engine. A specific embodiment of this system is described below.

[0682] System configuration

[0683] The system consists of the following major components:

[0684] 1. Smart home assistant: A device that receives voice and text queries from the user.

[0685] 2. Server: The central system for data processing and analysis.

[0686] 3. Generative AI: Artificial intelligence that analyzes user behavior patterns and emotions and generates instructions for controlling home appliances.

[0687] 4. Emotion engine: A system that analyzes emotions from the user's facial expressions and tone of voice.

[0688] 5. Smart Meter: A device that monitors electricity usage in the home in real time.

[0689] 6. Music player: A device that plays music based on instructions from generative AI.

[0690] Program processing

[0691] Electricity usage inquiries

[0692] 1. A user asks their smart home assistant via voice or text, "How much electricity have I used this month?"

[0693] 2. The device (smart home assistant) receives this query and sends a data request to the server.

[0694] 3. The server collects electricity usage data from the smart meter, analyzes it, and sends it to the device.

[0695] 4. The device responds to the user, "This month's electricity usage is 200 kWh."

[0696] Automated home appliance control

[0697] 1. The emotion engine analyzes the user's facial expressions and tone of voice and recognizes that the user is happy.

[0698] 2. The server receives the user's emotion information from the emotion engine.

[0699] 3. The server requests the generative AI to issue instructions for controlling home appliances based on the user's emotions.

[0700] 4. The generative AI generates instructions for controlling home appliances based on the user's emotions. For example, if the user is happy, it generates an instruction to a music player to "play the user's favorite music."

[0701] 5. The server sends instructions from the generative AI to the music playback device.

[0702] 6. The music player receives the command and plays the music.

[0703] Specific examples

[0704] A user asks their smart home assistant, "How much electricity have I used this month?"

[0705] The terminal receives this inquiry and sends a data request to the server.

[0706] The server obtains the data "200 kWh" from the smart meter, analyzes it, and sends it to the terminal.

[0707] The device responds to the user, "This month's electricity usage is 200 kWh."

[0708] The emotion engine recognizes that the user is happy based on their facial expression and tone of voice.

[0709] The server requests the generative AI to issue instructions to the music playback device.

[0710] The generative AI generates an instruction such as "Play the user's favorite music," and the server sends it to the music playback device.

[0711] The music player receives the instruction and plays the music.

[0712] Prompt Sentence Examples

[0713] "Tell me how much electricity I used this month."

[0714] "Set music to play when the user is happy"

[0715] This system allows the user to easily check the amount of electricity used and enjoy a comfortable living space that suits their emotions. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0716] Electricity usage inquiries

[0717] Step 1:

[0718] A user asks their smart home assistant via voice or text, "How much electricity did I use this month?"

[0719] Input: User's voice or text query

[0720] Output: Query request to smart home assistant

[0721] Step 2:

[0722] The device (smart home assistant) receives this query and sends a data request to the server.

[0723] Input: User inquiry request

[0724] Data processing: Analyzes the query and generates a data request to send to the server

[0725] Output: Data request to the server

[0726] Step 3:

[0727] The server obtains electricity usage data from the smart meter.

[0728] Input: Data request from terminal

[0729] Data processing: Obtaining electricity usage data from smart meters

[0730] Output: Acquired electricity usage data

[0731] Step 4:

[0732] The server analyzes the acquired data and generates information to provide to the user.

[0733] Input: Acquired electricity usage data

[0734] Data calculation: Analysis of electricity usage data

[0735] Output: Analysis result (e.g. 200 kWh)

[0736] Step 5:

[0737] The server sends the analysis results to the device.

[0738] Input: Analysis results

[0739] Data processing: Converting analysis results into a data format for sending to the terminal

[0740] Output: Sending analysis results to the device

[0741] Step 6:

[0742] The device responds to the user by voice or text, saying, "This month's electricity usage is 200 kWh."

[0743] Input: Analysis results from the server

[0744] Data processing: Converting analysis results into audio or text to communicate to users

[0745] Output: Answer to the user

[0746] Automated home appliance control

[0747] Step 1:

[0748] The emotion engine analyzes the user's facial expressions and tone of voice and recognizes that the user is happy.

[0749] Input: User facial expressions and tone of voice

[0750] Data Computing: Sentiment Analysis

[0751] Output: User's emotional state (e.g., happy)

[0752] Step 2:

[0753] The server receives the user's emotion information from the emotion engine.

[0754] Input: Emotion information from the emotion engine

[0755] Output: Receiving emotional information

[0756] Step 3:

[0757] The server requests the generative AI to issue instructions for controlling home appliances based on the user's emotions.

[0758] Input: Emotion information

[0759] Data processing: Generate requests to generate instructions for controlling home appliances based on emotional information

[0760] Output: Requirements for generative AI

[0761] Step 4:

[0762] Generative AI generates instructions for controlling home appliances based on the user's emotions.

[0763] Input: Request from server

[0764] Data calculation: Generating instructions for controlling home appliances based on emotional information

[0765] Output: Instructions for controlling home appliances (e.g., instructing a music player to "play the user's favorite music")

[0766] Step 5:

[0767] The server sends instructions from the generative AI to the music playback device.

[0768] Input: Instructions for controlling home appliances from generative AI

[0769] Data processing: Converting instructions into a data format for sending to a music playback device

[0770] Output: Sending instructions to a music player

[0771] Step 6:

[0772] The music player receives the instruction and plays the music.

[0773] Input: Instructions from the server

[0774] Data processing: Play music based on instructions

[0775] Output: Music playback

[0776] In this way, the user can easily check the amount of electricity used and can enjoy a comfortable living space according to their feelings.

[0777] (Application example 3)

[0778] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0779] Conventional self-driving vehicles do not adjust the in-car environment according to passengers' emotions, and there is a lack of personalized services to make passengers comfortable. Furthermore, there are limited functions for monitoring and querying electricity usage, making it difficult for passengers to understand the vehicle's status in real time. This leads to lower passenger satisfaction and makes it difficult to provide a comfortable travel experience.

[0780] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0781] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for monitoring electricity usage, means for inquiring about electricity usage by voice or text, means for recognizing user emotions using an emotion recognition engine, and means for automatically adjusting the in-car environment according to the user's emotions. This makes it possible to provide a personalized in-car environment according to the passenger's emotions, resulting in a comfortable travel experience.

[0782] "Generative AI" is artificial intelligence that analyzes a user's emotions and behavioral patterns and provides appropriate responses and control based on that.

[0783] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[0784] A "smart home assistant" is a system that receives user instructions via voice or text and controls home appliances and other smart devices.

[0785] "Automatic home appliance control" is a function that automatically operates home appliances based on user instructions and behavioral patterns.

[0786] "Electricity usage monitoring" is a function that monitors the electricity usage of specific time periods or specific home appliances in real time, and collects and analyzes the data.

[0787] "Means to inquire about electricity usage by voice or text" is a function that allows users to inquire about information about electricity usage by voice or text.

[0788] An "emotion recognition engine" is a system that analyzes the user's emotions from their facial expressions and voice, and uses that information to provide appropriate responses and control.

[0789] "Means for automatically adjusting the in-car environment according to the user's emotions" is a function that automatically adjusts environmental settings such as lighting, music, and temperature in the car based on information obtained from the emotion recognition engine.

[0790] As an embodiment of the present invention, a system for recognizing the emotions of passengers in an autonomous vehicle and automatically adjusting the in-vehicle environment accordingly is provided. Specific embodiments of this system are described below.

[0791] System configuration

[0792] This system is configured using the following hardware and software.

[0793] 1. Hardware:

[0794] Camera: Used to capture passengers' facial expressions.

[0795] Microphone: Used to capture passenger voices.

[0796] Speaker: Used for voice response and music playback.

[0797] Display: Used to display text information.

[0798] 2. Software:

[0799] Emotion recognition engine: Used to recognize emotions from facial expressions and voice (e.g., emotion recognition service).

[0800] Generative AI: Used to adjust the in-car environment based on information from emotion recognition engines (e.g., generative AI models).

[0801] Smart home assistants: Used to respond to voice and text inquiries (e.g., voice assistants).

[0802] Data processing and calculation

[0803] 1. Emotion recognition:

[0804] Data obtained from the camera and microphone is input into an emotion recognition engine to analyze passenger emotions.

[0805] The emotion recognition engine analyzes facial expressions and voice characteristics to recognize passengers' emotions, such as "happy" or "relaxed."

[0806] 2. Generative AI prompt generation:

[0807] Based on the emotional information obtained from the emotion recognition engine, the generative AI generates appropriate prompts.

[0808] For example, if the passenger is happy, generate a prompt saying "If the passenger is happy, play their favorite music."

[0809] 3. Smart home assistant response:

[0810] Based on the generated prompts, the smart home assistant generates a response via voice or text.

[0811] For example, in response to the query, "How much electricity did you use this month?", the smart home assistant will respond, "Your electricity usage this month is 100kWh."

[0812] Specific examples

[0813] 1. Examples of emotion recognition:

[0814] The camera captures passengers' smiles and the microphone captures their laughter.

[0815] An emotion recognition engine analyzes this data and determines that the passenger is "happy."

[0816] 2. Examples of prompts:

[0817] "If passengers are happy, play their favorite music."

[0818] "If passengers want to relax, adjust the lighting to a warmer color."

[0819] This system makes it possible to provide a personalized in-car environment that responds to passenger emotions, resulting in a comfortable travel experience.

[0820] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0821] Step 1:

[0822] Capturing Emotional Data

[0823] The server captures passengers' facial expressions and voices using a camera and microphone. The video data from the camera and the audio data from the microphone are input. These data are sent to an emotion recognition engine.

[0824] Step 2:

[0825] emotion recognition

[0826] The server analyzes the captured facial and audio data using an emotion recognition engine. It extracts facial features and tone of voice from the input video and audio data to recognize the passenger's emotions. For example, if a smile or laughter is detected, it is recognized as "happy." The recognized emotional information is generated as output.

[0827] Step 3:

[0828] Prompt Generation

[0829] The server uses generative AI to generate prompts based on the emotional information obtained from the emotion recognition engine. Based on the input emotional information, it generates a prompt sentence containing an appropriate response or control instruction. For example, a generated prompt might say, "If the passenger is happy, play their favorite music." The generated prompt sentence is obtained as the output.

[0830] Step 4:

[0831] Implementing environmental adjustment instructions

[0832] The server sends the generated prompt sentence to the smart home assistant. The smart home assistant executes an instruction to adjust the in-car environment based on the prompt sentence. For example, it sends an instruction to a music player to "play your favorite music." As an output, the in-car environment is adjusted.

[0833] Step 5:

[0834] Voice and text response

[0835] When a user inquires about electricity usage by voice or text, the server receives the inquiry through the smart home assistant. The server analyzes the input voice or text data to obtain information about electricity usage. For example, in response to the inquiry "How much electricity was used this month?", the server generates a response such as "This month's electricity usage is 100kWh." The response is obtained as voice or text as output.

[0836] In this way, through a series of processing steps, the server provides a personalized in-car environment according to the passenger's emotions and responds to inquiries about electricity usage.

[0837] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0838] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / OpenAI.com / blog / ChatGPT> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0839] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0840] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0841] [Second embodiment]

[0842] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0843] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0844] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0845] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0846] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0847] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0848] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0849] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0850] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0851] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0852] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0853] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0854] "Example 1"

[0855] In one embodiment of the present invention, generative AI learns user behavior patterns and automates home appliance control based on the results of that learning. For example, if it learns that the user wakes up at 7:00 every morning, it will open the curtains and turn on the coffee maker at that time.

[0856] "Example 2"

[0857] In one embodiment of the present invention, the electricity usage monitoring means analyzes in detail the electricity usage of specific time periods or specific home appliances and makes energy saving suggestions. For example, the means analyzes the time periods when an air conditioner is running or when a microwave oven is used, and presents suggestions to the user on optimizing the usage time and frequency of those home appliances.

[0858] "Example 3"

[0859] Furthermore, as one embodiment of the present invention, a means for inquiring about electricity usage by voice or text is provided. A user can inquire of the smart home assistant, such as "How much electricity have I used this month?", and the smart home assistant will respond to the inquiry based on the electricity usage information obtained from the monitoring means.

[0860] The processing flow of each embodiment will be described below.

[0861] "Example 1"

[0862] Step 1: The generative AI learns the user's behavioral patterns. For example, it learns that the user wakes up at 7am every morning.

[0863] Step 2: Based on the learned behavioral patterns, the generative AI automates home appliance control, for example, automatically opening the curtains at 7:00 AM, the time when the user wakes up.

[0864] Step 3: Similarly, it also controls the coffee maker to start automatically, so that the user can enjoy a cup of coffee as soon as they wake up.

[0865] "Example 2"

[0866] Step 1: The electricity usage monitoring method analyzes in detail the electricity usage of specific time periods or specific home appliances, for example, analyzing the time periods when the air conditioner is running.

[0867] Step 2: Based on the analysis results, the electricity usage monitoring tool will make energy-saving suggestions to the user, such as optimizing the usage time and frequency of the air conditioner.

[0868] "Example 3"

[0869] Step 1: The user makes a voice or text inquiry to the smart home assistant about their electricity usage, such as "How much electricity did I use this month?"

[0870] Step 2: The smart home assistant responds to the query based on the electricity usage information obtained from the monitoring method, allowing the user to easily understand their electricity usage.

[0871] Example 1

[0872] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0873] Conventional smart home systems require users to manually operate home appliances, limiting the scope of automation. Furthermore, there is a lack of monitoring of electricity usage and suggestions for saving energy, making it difficult to improve energy efficiency. Furthermore, there is no way to control home appliances based on user behavior patterns, resulting in low user convenience.

[0874] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0875] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for monitoring electricity usage, means for inquiring about electricity usage by voice or text, means for collecting user behavioral data, means for analyzing and learning from the behavioral data, means for generating a home appliance control schedule based on the learning results, and means for controlling home appliances based on the home appliance control schedule. This enables automatic home appliance control based on the user's behavioral patterns, thereby improving energy efficiency and user convenience.

[0876] "Generative AI" is an artificial intelligence technology that learns user behavior patterns and automates home appliance control based on the results of that learning.

[0877] A "smart home appliance control app" is application software that allows you to remotely control and manage home appliances using devices such as smartphones and tablets.

[0878] A "smart home assistant" is a digital assistant that uses voice recognition and natural language processing to operate home appliances according to user instructions.

[0879] "Automatic home appliance control" is a function that automatically operates home appliances based on the user's behavioral patterns and schedules.

[0880] "Electricity usage monitoring" is a function that monitors and records the power consumption of home appliances and the entire system in real time.

[0881] The "means for inquiring about electricity usage by voice or text" is an interface that allows users to check current electricity consumption status and past usage history using voice commands or text input.

[0882] "Means for collecting user behavioral data" refers to a function that uses smart home devices and sensors to record user behavior and environmental information.

[0883] "Means for analyzing and learning from behavioral data" refers to algorithms and artificial intelligence technologies that analyze collected user behavioral data and find patterns and regularities.

[0884] The "means for generating a home appliance control schedule based on the learning results" is a function that determines the optimal timing and method for operating home appliances based on the analyzed and learned user behavior patterns.

[0885] The "means for controlling home appliances based on the home appliance control schedule" refers to an interface or control system for actually operating home appliances according to the generated schedule.

[0886] MODE FOR CARRYING OUT THE INVENTION

[0887] This invention relates to a system that uses generative AI to learn user behavior patterns and automates home appliance control based on the learning results. Specific embodiments of this system are described below.

[0888] 1. Program Generation

[0889] The server uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate a program that learns user behavior patterns. This program has the function of collecting and analyzing user behavior data and generating a home appliance control schedule based on the learning results.

[0890] 2. Program Processing

[0891] The server collects user behavior data from smart home devices (e.g., smart speakers, smart lights, smart plugs, etc.), including the time and frequency with which users interact with the devices and data from environmental sensors (e.g., temperature, light intensity, motion detection, etc.).

[0892] The server then inputs the collected behavioral data into a generative AI model, which analyzes and learns the user's behavioral patterns. The AI ​​model then finds patterns in this data and identifies the user's typical behavioral schedule.

[0893] The server then generates a schedule for controlling home appliances based on the learning results. For example, the server might send an instruction to the smart curtain device to open the curtains at 7:00 every morning, and an instruction to the smart light device to turn off the lights at 10:00 every night.

[0894] Finally, the terminal (smart home device) receives instructions from the server and actually controls the home appliances, thereby realizing automation that matches the user's behavioral patterns.

[0895] 3. Specific Examples

[0896] As a concrete example, consider a case where a user has the habit of going to bed at 10 p.m. every night. In this case, the server will set the system to automatically turn off the lights and switch the air conditioner to energy-saving mode at 10 p.m.

[0897] Example prompt sentence:

[0898] Design a system that learns user behavior patterns and automates home appliance control. For example, if the user wakes up at 7:00 every morning, design the system to open the curtains and turn on the coffee maker at that time.

[0899] In this way, the server utilizes generative AI models to realize automated home appliance control to make users' lives more comfortable.

[0900] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0901] Program processing flow

[0902] Step 1: Collect user behavior data

[0903] The server collects user behavior data from smart home devices (e.g., smart speakers, smart lights, smart plugs, etc.), specifically, the time and frequency at which users operate the devices, and data from environmental sensors (e.g., temperature, illuminance, motion detection, etc.).

[0904] Input: Operation logs from smart home devices, environmental sensor data

[0905] Data processing: Organize and save operation logs and sensor data as time-series data

[0906] Output: Organized behavioral data

[0907] Specific behavior:

[0908] The server records the time when the user says "good morning" from the smart speaker.

[0909] The server records the on / off times of the smart lights.

[0910] Step 2: Analyze and learn from behavioral data

[0911] The server inputs the collected behavioral data into a generative AI model (e.g., GPT-4 (registered trademark)) to analyze and learn the user's behavioral patterns. The AI ​​model finds regularities and patterns in this data and identifies the user's typical behavioral schedule.

[0912] Input: Organized behavioral data

[0913] Data Computation: Using generative AI models to analyze behavioral data and learn patterns

[0914] Output: User behavior patterns

[0915] Specific behavior:

[0916] The server learns the pattern of users saying "good morning" every morning at 7am.

[0917] The server learns the pattern that the smart lights turn off at 10pm every night.

[0918] Step 3: Generate a home appliance control schedule

[0919] The server generates a schedule for controlling home appliances based on the learning results, which includes instructions for operating specific devices at specific times.

[0920] Input: User behavior patterns

[0921] Data processing: Determine the optimal timing and method for operating home appliances based on behavioral patterns

[0922] Output: Home appliance control schedule

[0923] Specific behavior:

[0924] The server sends an instruction to the smart curtain device to open the curtains at 7am every morning.

[0925] The server sends an instruction to the smart lights to turn off the lights at 10pm every night.

[0926] Step 4: Implementing appliance control

[0927] The terminals (smart home devices) receive instructions from the server and actually control the home appliances, thereby realizing automation that matches the user's behavioral patterns.

[0928] Input: Home appliance control schedule

[0929] Data calculation: Operate home appliances according to a schedule

[0930] Output: Actual controlled behavior of the appliance

[0931] Specific behavior:

[0932] The device receives instructions from the server and opens the curtains at 7:00 every morning.

[0933] The terminal receives instructions from the server to turn off the lights at 10 p.m. every night.

[0934] (Application example 1)

[0935] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0936] Conventional smart home appliance control systems are specialized for controlling home appliances in the home, and do not support automatic control of equipment in commercial facilities such as brick-and-mortar stores. Furthermore, manually controlling in-store equipment places a heavy burden on store staff, hindering efficient store operations. Furthermore, improper control of in-store equipment can result in energy waste and reduced customer satisfaction. To solve these issues, a system that can learn user behavior patterns and automatically control equipment is needed, even in brick-and-mortar stores.

[0937] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0938] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for learning user behavior patterns and automatically controlling in-store equipment, and means for controlling in-store equipment. This enables automatic control of equipment based on user behavior patterns even in physical stores, reducing the burden on store staff and enabling efficient store management.

[0939] "Generative AI" is a type of artificial intelligence that learns user behavior patterns and automatically controls itself based on the results of that learning.

[0940] A "smart home appliance control app" is an application that runs on mobile devices such as smartphones and tablets and allows remote control of home appliances.

[0941] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to control home appliances and facilities.

[0942] "Automatic home appliance control" is a function that automatically operates home appliances based on user behavior patterns and predictions.

[0943] "Electricity usage monitoring" is a function that monitors and records the electricity consumption of specific time periods or specific home appliances in real time.

[0944] "A means for inquiring about electricity usage via voice or text" refers to an interface that allows users to check their current electricity consumption status using voice commands or text input.

[0945] "Means of learning user behavior patterns and automatically controlling in-store equipment" refers to a function that analyzes the behavior of store staff and customers and automatically operates store equipment such as lighting, music, and air conditioning based on the results.

[0946] "Means for controlling in-store equipment" refers to systems or devices for manually or automatically operating in-store equipment such as lighting, music, and air conditioning.

[0947] A system for implementing this invention includes generative AI, a smart home appliance control app, a smart home assistant, a means for monitoring electricity usage, a means for inquiring about electricity usage via voice or text, a means for learning user behavior patterns and automatically controlling in-store equipment, and a means for controlling in-store equipment.

[0948] System Program

[0949] The system's program is implemented using programming languages ​​such as Python. The generative AI learns user behavior patterns and automatically controls in-store equipment based on the results of that learning. Specifically, it performs the following processes:

[0950] Processing Description

[0951] 1. Generative AI learning phase

[0952] The server collects and analyzes past behavioral data to learn user behavior patterns, including when store staff open the store and when they operate specific equipment, and uses machine learning algorithms to model these patterns.

[0953] 2. Predicting behavioral patterns

[0954] The server predicts future behavior based on learned behavioral patterns. For example, if it learns that store staff open the store at 9:00 a.m. every morning, it will automatically control the store's equipment to match that time.

[0955] 3. Automatic control of equipment

[0956] Based on predicted behavioral patterns, the server automatically controls in-store equipment, such as turning on lights, playing music, and setting the air conditioning to an appropriate temperature.

[0957] 4. Monitoring electricity usage

[0958] The server monitors and records electricity usage during specific time periods and for specific equipment in real time, preventing energy waste and enabling efficient energy management.

[0959] 5. Voice and text inquiries

[0960] Users can check the current power consumption status using voice commands or text input, allowing them to understand power consumption in real time and operate the equipment as needed.

[0961] Hardware and software used

[0962] Hardware: Servers, smartphones, tablets, smart home assistant devices

[0963] Software: Python, machine learning libraries (e.g., TENSORFLOW (registered trademark), scikit-learn (registered trademark)), smart home appliance control app

[0964] Specific examples

[0965] For example, by inputting the following prompt sentence into a generative AI model, it can learn the behavioral patterns of store clerks.

[0966] Learn the behavioral patterns of store staff who open the store at 9am every morning.

[0967] By using this prompt, the generative AI model can learn the behavioral patterns of store staff and automatically control in-store equipment based on the results, thereby reducing the burden on store staff and enabling efficient store operations.

[0968] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0969] Step 1:

[0970] The server collects user behavior data. Specifically, it obtains data such as when store staff open the store and when specific equipment is operated from smart home appliance control apps and smart home assistants. The input is past behavior data, and the output is a list of collected behavior data.

[0971] Step 2:

[0972] The server analyzes the collected behavioral data and learns the user's behavioral patterns. Specifically, it uses a machine learning algorithm to model the behavioral patterns. The input is a list of collected behavioral data, and the output is a learned behavioral pattern model.

[0973] Step 3:

[0974] The server predicts future behavior based on the learned behavioral patterns. Specifically, it uses a behavioral pattern model to predict what behavior will occur at a specific time. The input is the learned behavioral pattern model, and the output is a list of predicted behaviors.

[0975] Step 4:

[0976] The server automatically controls the equipment in the store based on the predicted behavior. Specifically, it performs operations such as turning on the lights, playing music, and setting the air conditioning to an appropriate temperature. The input is a list of predicted behaviors, and the output is the controlled status of the equipment.

[0977] Step 5:

[0978] The server monitors and records the electricity usage of specific time periods and specific equipment in real time. Specifically, it obtains power consumption data from smart home appliance control apps and smart home assistants and stores it in a database. The input is the power consumption data, and the output is the recorded power consumption data.

[0979] Step 6:

[0980] Users can check the current power consumption status using voice commands or text input. Specifically, they query the server through a smart home appliance control app or smart home assistant. The input is a voice command or text input, and the output is information about the current power consumption status.

[0981] Example 2

[0982] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0983] Conventional household electricity usage monitoring systems do not adequately collect and analyze electricity usage data, making it difficult to provide specific energy-saving suggestions to users. They also lack the functionality to learn user behavior patterns and automate home appliance control based on predictions. This has led to the issue of being unable to achieve efficient power management and energy saving.

[0984] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for linking with the generative artificial intelligence, the home appliance control application, and the home assistant, a means for realizing automatic home appliance control, a means for realizing electricity usage monitoring, a means for inquiring about electricity usage by voice or text, a means for collecting electricity usage data, a means for storing the collected data, a means for analyzing the stored data, a means for generating power-saving suggestions based on the analysis results, and a means for presenting the generated power-saving suggestions to the user. This enables detailed monitoring and analysis of electricity usage and provides specific power-saving suggestions to the user. Furthermore, efficient power management and power saving can be achieved by learning user behavior patterns and automating home appliance control based on predictions.

[0985] "Generative AI" is an AI technology that learns user behavior patterns and generates home appliance control and energy-saving suggestions based on predictions.

[0986] A "home appliance control application" is software for controlling home appliances through devices such as smartphones and tablets.

[0987] A "home assistant" is a device or software that interacts with users through voice or text to control home appliances and provide information.

[0988] "Automatic home appliance control" is a function that automatically adjusts the on / off and settings of home appliances based on user behavior patterns and predictions.

[0989] "Electricity usage monitoring" is the process of recording the electricity usage of each household appliance in real time and collecting it as data.

[0990] "A means of inquiring about electricity usage via voice or text" refers to an interface that allows users to check their current electricity usage and past usage history through voice commands or text input.

[0991] The "means of collecting electricity usage data" refers to a system that uses smart meters and IoT devices to obtain real-time electricity usage data for each household appliance.

[0992] "Means for storing collected data" refers to a function that stores collected electricity usage data in a database so that it can be analyzed and referenced later.

[0993] "Means for analyzing stored data" refers to software or algorithms for analyzing stored electricity usage data and detecting usage patterns or anomalies.

[0994] The "means for generating energy-saving suggestions based on the analysis results" refers to a generative artificial intelligence that suggests specific energy-saving methods to users based on the results of data analysis.

[0995] The "means for presenting the generated power saving suggestions to the user" is an interface that notifies the user's terminal of the generated power saving suggestions and presents them visually or audibly.

[0996] The present invention is a system for monitoring the amount of electricity used in a home in detail and providing specific energy-saving suggestions to the user. A specific embodiment of this system will be described below.

[0997] The server first collects real-time electricity usage data from each household appliance using smart meters and IoT devices. These devices record the usage time and power consumption of appliances such as air conditioners and microwave ovens. For example, if the air conditioner is running, it records that the power consumption is 1.5 kWh per hour.

[0998] The server then stores the collected electricity usage data in a database, which contains detailed information about each appliance, such as usage time, frequency, and power consumption. For example, the data might be recorded in the format "October 1, 2023, 10:00 - Air conditioner - 1.5kWh."

[0999] The server analyzes the stored data using data analysis software such as Python and R. Specifically, it extracts electricity usage for specific time periods and for specific home appliances and analyzes usage patterns. For example, it confirms that air conditioner usage is concentrated between 9:00 AM and 5:00 PM.

[1000] The server then uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate power-saving suggestions based on the results of the data analysis. The following prompt is input to the generative AI model: "Generate power-saving suggestions for the user based on the following data. Data: Air conditioner usage time: 9:00 AM to 5:00 PM, Usage frequency: 5 times a day. Suggestion: How to optimize air conditioner usage time."

[1001] The generative AI model generates a suggestion such as, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM. We especially recommend refraining from using it in the morning when the outside temperature is relatively low."

[1002] Finally, the server sends the generated energy-saving suggestions to the user's device. The user can check the suggestions via their smartphone or computer. For example, a notification will be displayed saying, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM."

[1003] In this way, the server can monitor and analyze electricity usage in detail and provide specific energy-saving suggestions to users. It can also learn user behavior patterns and automate home appliance control based on predictions, achieving efficient power management and energy savings.

[1004] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1005] Step 1: Data collection

[1006] The server collects real-time electricity usage data from each home appliance via smart meters and IoT devices. As input, it provides data on the usage time and power consumption of each appliance. For example, if an air conditioner is running, it records that the power consumption is 1.5 kWh per hour. As output, it obtains the collected electricity usage data.

[1007] Step 2: Save data

[1008] The server stores the collected electricity usage data in a database. As input, the data collected in step 1 is provided. For example, the data is recorded in the format "October 1, 2023 10:00 - Air Conditioner - 1.5kWh". As output, the electricity usage data stored in the database is obtained.

[1009] Step 3: Data analysis

[1010] The server uses data analysis software such as Python or R to analyze the stored data. The electricity usage data stored in the database is provided as input. Specifically, the server extracts electricity usage for specific time periods and for specific home appliances and analyzes usage patterns. For example, it can confirm that air conditioner usage is concentrated between 9:00 AM and 5:00 PM. The analysis results are obtained as output.

[1011] Step 4: Generate power saving suggestions

[1012] The server uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate power-saving suggestions based on the results of the data analysis. The analysis results obtained in step 3 are provided as input. The following prompt sentence is input to the generative AI model: "Please generate power-saving suggestions for the user based on the following data. Data: Air conditioner usage time: 9:00 AM to 5:00 PM, usage frequency: 5 times a day. Suggestion: How to optimize air conditioner usage time." The generated power-saving suggestions are obtained as output.

[1013] Step 5: Present your proposal

[1014] The server sends the generated power-saving suggestions to the user's device. As input, the power-saving suggestions generated in step 4 are provided. The user can check the suggestions through their smartphone or computer. For example, a notification may be displayed saying, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM." As output, the power-saving suggestions presented to the user are obtained.

[1015] (Application example 2)

[1016] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1017] Conventional electricity usage monitoring systems are often only applied to household appliances, making it difficult to optimize electricity usage and propose energy-saving measures in physical stores. In addition, there was a lack of concrete means for store managers to grasp electricity usage in real time and take appropriate energy-saving measures. As a result, the efficiency of electricity consumption in physical stores was not fully achieved.

[1018] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for linking the generative AI with a smart home appliance control app and a smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for monitoring the electricity usage of each home appliance in the store in real time, means for analyzing usage patterns of specific time periods and specific home appliances, means for notifying the store manager of energy-saving suggestions, and means for displaying past energy usage data in graphs to visualize the energy-saving effects. This enables optimization of electricity usage in physical stores and efficient energy-saving measures.

[1019] "Generative AI" is a type of artificial intelligence that generates new information and predictions based on data.

[1020] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[1021] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to comprehensively manage and control home appliances and systems.

[1022] "Automatic home appliance control" is a function that automatically operates and manages home appliances based on user settings and behavioral patterns.

[1023] "Electricity usage monitoring" is a function that measures and records the electricity consumed by home appliances and systems in real time.

[1024] "Voice and text electricity usage inquiry means" refers to an interface that allows a user to obtain electricity usage information through voice commands or text input.

[1025] "Means for monitoring the electricity usage of each home appliance in the store in real time" is a function that measures the electricity consumption of home appliances installed in a physical store in real time and collects data.

[1026] "Means for analyzing usage patterns during specific time periods or for specific home appliances" refers to a function that analyzes usage trends during specific time periods or for specific home appliances based on collected electricity consumption data.

[1027] The "means for notifying the store manager of power saving suggestions" is a function that notifies the store manager of methods for reducing power consumption and optimal usage patterns based on the analysis results.

[1028] "A means of displaying past electricity usage data in a graph and visualizing the effects of energy saving" is a function that visually displays power consumption data collected in the past and clearly shows the effects of energy saving.

[1029] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[1030] System Configuration

[1031] This system uses the following hardware and software:

[1032] Hardware:

[1033] Smart meter: Measures the electricity usage of each appliance in the store in real time.

[1034] Server: Collects, stores, and analyzes data.

[1035] Smartphone or tablet: A device used by the store manager to operate the application.

[1036] software:

[1037] Python: A programming language for data analysis and visualization.

[1038] Pandas: A library for data preprocessing and aggregation.

[1039] Matplotlib: A library for generating graphs.

[1040] Smart home appliance control app: An application for remotely controlling home appliances.

[1041] Smart home assistant: Software that manages and controls home appliances and systems in an integrated manner.

[1042] Data processing and calculation

[1043] The server receives electricity usage data collected from smart meters in real time and stores it in a database. Next, it preprocesses the data using Pandas to analyze usage patterns for specific time periods and specific appliances. The analysis results are visualized as graphs using Matplotlib.

[1044] Generative AI learns user behavior patterns from collected data, automates home appliance control based on predictions, and generates energy-saving suggestions that are notified to store managers via a smart home appliance control app.

[1045] Specific examples

[1046] For example, if air conditioner usage peaks at 2 p.m., the generative AI will generate a power-saving suggestion such as, "Air conditioner usage peaks at 2 p.m. Consider shifting usage to the morning." This suggestion is then notified to the store manager via the smart home appliance control app.

[1047] Example prompt sentence:

[1048] "Peak air conditioning usage time is 2 p.m. Consider shifting usage to the morning."

[1049] In this way, it becomes possible to optimize electricity usage in physical stores and take efficient energy-saving measures.

[1050] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1051] Step 1:

[1052] The server receives real-time electricity usage data from the smart meter. The input is the electricity usage data from the smart meter, and the output is the raw data stored in the database in the server. Specifically, the server receives data packets sent from the smart meter and records them in the database.

[1053] Step 2:

[1054] The server uses Pandas to preprocess the received electricity usage data. The input is the raw data saved in step 1, and the output is the preprocessed clean data. Specifically, the server complements missing values ​​in the data and aligns the data based on timestamps.

[1055] Step 3:

[1056] The server analyzes the usage patterns of specific time periods and specific appliances based on the preprocessed data. The input is the clean data obtained in step 2, and the output is the analysis results of the usage patterns. Specifically, the server uses Pandas to group the data and aggregate the usage of each appliance by time period.

[1057] Step 4:

[1058] The server uses generative AI to learn user behavior patterns and automates home appliance control based on predictions. The input is the analysis results of usage patterns obtained in step 3, and the output is the automated home appliance control settings. Specifically, the server uses a machine learning algorithm to model behavior patterns and generate an optimal home appliance control schedule.

[1059] Step 5:

[1060] The server generates power-saving suggestions using generative AI and notifies the store manager through the smart home appliance control app. The input is the automated home appliance control settings obtained in step 4, and the output is a notification of the power-saving suggestions. Specifically, the server generates the power-saving suggestions using the generative AI model and sends them to the app as prompts.

[1061] Step 6:

[1062] The terminal (smartphone or tablet) receives the power-saving suggestion sent from the server and notifies the store manager. The input is the notification of the power-saving suggestion generated in step 5, and the output is a notification to the store manager. Specifically, the terminal displays the power-saving suggestion using the push notification function.

[1063] Step 7:

[1064] The user (store manager) checks the power-saving suggestions through the terminal and adjusts the appliance usage schedule as necessary. The input is the power-saving suggestions received in step 6, and the output is the adjusted appliance usage schedule. Specifically, the user operates the app interface to change the appliance settings.

[1065] Step 8:

[1066] The server displays past electricity usage data in a graph to visualize the power saving effect. The input is the clean data obtained in step 2 and the analysis results of the usage pattern obtained in step 3, and the output is the power saving effect displayed as a graph. Specifically, the server uses Matplotlib to visualize the data and generate the graph.

[1067] Example 3

[1068] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1069] In conventional smart home systems, users had to manually open a dedicated application to check their electricity usage. Furthermore, when using a voice assistant to inquire about electricity usage, it was difficult to obtain accurate data or generate natural responses. Furthermore, there was a lack of technology to accurately understand the user's intent and provide appropriate responses. This resulted in a poor user experience and impaired the convenience of smart home systems.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1071] In this invention, the server includes means for converting voice to text, means for analyzing the text and understanding the user's intent, means for acquiring electricity usage data, means for generating a response based on the acquired data, and means for conveying the generated response to the user by voice. This makes it possible to acquire accurate data and generate a natural response when a user inquires about electricity usage by voice or text.

[1072] "Generative AI" is an artificial intelligence technology that analyzes user input and generates answers and predictions in natural language.

[1073] A "smart home appliance control app" is application software that allows you to remotely control home appliances using devices such as smartphones and tablets.

[1074] A "smart home assistant" is a device that uses voice recognition and natural language processing to respond to users' voice commands and control home appliances and systems.

[1075] "Automatic home appliance control" is a function that automatically operates home appliances based on user settings and behavior patterns.

[1076] "Electricity usage monitoring" is a function that monitors and records the amount of electricity consumed within the home in real time.

[1077] A "speech-to-text means" is a technology for converting a user's voice input into textual information.

[1078] "Means for analyzing text and understanding user intent" refers to technology that analyzes text data to understand what the user is looking for.

[1079] "Means for acquiring electricity usage data" refers to a technique for collecting electricity consumption data from the monitoring system.

[1080] "Means for generating answers based on acquired data" refers to technology for analyzing collected data and generating appropriate answers to user inquiries.

[1081] The "means for conveying the generated answer to the user by voice" is a technology for converting the text-format answer into voice and conveying it to the user.

[1082] MODE FOR CARRYING OUT THE INVENTION

[1083] The present invention relates to a smart home system that allows users to inquire about electricity usage by voice or text. Specific embodiments of this system will be described below.

[1084] System configuration

[1085] The system consists of the following major components:

[1086] 1. Smart Home Assistant (Device)

[1087] 2. Server

[1088] 3. Monitoring System

[1089] Hardware and software used

[1090] Speech Recognition Software: Google® Cloud Speech-to-Text API

[1091] Generative AI model: GPT-4(registered trademark) by OpenAI(registered trademark)

[1092] Text-to-speech software: Amazon Polly or Google® Cloud Text-to-Speech

[1093] Monitoring systems: smart meters and energy monitoring systems

[1094] Program processing

[1095] When a user asks the smart home assistant, "How much electricity have I used this month?", the following process takes place:

[1096] 1. Speech-to-text

[1097] The device records the user's voice and sends the voice data to the Google® Cloud Speech-to-Text API, which converts the voice into text and returns it to the device.

[1098] 2. Sending a text

[1099] The terminal sends the converted text to the server. The text data is sent to the server as an HTTP request.

[1100] 3. Text Analysis

[1101] The server analyzes the received text and uses a generative AI model (OpenAI's GPT-4®) to understand the user's intent. The generative AI model generates an appropriate prompt.

[1102] 4. Obtaining electricity usage data

[1103] The server calls the monitoring system's API to retrieve electricity usage data, and may also retrieve past data from a database.

[1104] 5. Answer Generation

[1105] The server uses the electricity usage data to generate answers to user queries, again in natural language using a generative AI model.

[1106] 6. Sending answers and voice synthesis

[1107] The server sends the generated answer to the device, which then verbally communicates the answer to the user. The voice synthesis uses Amazon Polly and Google Cloud Text-to-Speech.

[1108] Specific examples

[1109] As a concrete example, consider a scenario in which a user asks a smart home assistant, "How much electricity have I used this month?" In this case, the following prompt sentence is input to the generative AI model:

[1110] Example prompt sentence:

[1111] A user asks, "How much electricity did I use this month?" Data obtained from a monitoring system shows that the electricity usage this month is 150 kWh. Use this information to generate a natural answer for the user.

[1112] Based on this prompt, the generative AI model generates the following answer:

[1113] "This month's electricity usage is 150 kWh."

[1114] This answer is sent to the terminal, which then uses speech synthesis software to communicate it to the user. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1115] Step 1:

[1116] The user asks the smart home assistant by voice, "How much electricity have I used this month?"

[1117] Input: User's voice

[1118] Output: Audio data

[1119] Specific action: The user speaks to the smart home assistant.

[1120] Step 2:

[1121] The device records the user's voice and sends the voice data to the Google® Cloud Speech-to-Text API, which converts the voice into text and returns it to the device.

[1122] Input: Audio data

[1123] Output: Text data

[1124] Specific operations: The device sends voice data to the Google® Cloud Speech-to-Text API and receives text data.

[1125] Step 3:

[1126] The terminal sends the converted text to the server.

[1127] Input: Text data

[1128] Output: Text data as an HTTP request

[1129] Specific operation: The device sends text data to the server as an HTTP request.

[1130] Step 4:

[1131] The server analyzes the received text and uses a generative AI model (OpenAI's GPT-4®) to understand the user's intent. The generative AI model generates an appropriate prompt.

[1132] Input: Text data

[1133] Output: prompt statement

[1134] Specific operation: The server inputs text data into the generative AI model and generates a prompt sentence.

[1135] Step 5:

[1136] The server calls the monitoring system's API and obtains electricity usage data.

[1137] Input: prompt statement

[1138] Output: Electricity usage data

[1139] Specific operation: The server calls the monitoring system's API and obtains electricity usage data.

[1140] Step 6:

[1141] The server uses the electricity usage data to generate answers to user queries, again in natural language using a generative AI model.

[1142] Input: Electricity usage data

[1143] Output: A natural language response

[1144] How it works: The server inputs electricity usage data into a generative AI model and generates a response in natural language.

[1145] Step 7:

[1146] The server sends the generated answer to the device, which then verbally communicates the answer to the user. The voice synthesis uses Amazon Polly and Google Cloud Text-to-Speech.

[1147] Input: Natural language response

[1148] Output: Audio data

[1149] Specific operations: The server sends the generated answer to the device as an HTTP response. The device inputs the received text into speech synthesis software to generate voice data. The device plays the generated voice to the user.

[1150] (Application example 3)

[1151] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1152] Conventional smart home systems monitor electricity usage within the home and provide functions that allow users to inquire about it by voice or text, but similar functions are also required in physical stores. In particular, there is a need for a system that monitors energy usage in physical stores in real time, allowing store staff to easily check energy usage and receive energy-saving advice. However, such a system does not currently exist, and energy management is not being carried out efficiently.

[1153] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1154] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for monitoring energy usage in the physical store in real time, means for store staff to inquire about energy usage by voice or text using a smartphone, and means for providing energy saving advice based on energy usage data. This enables efficient management of energy usage and power saving even in physical stores.

[1155] "Generative AI" is artificial intelligence that learns user behavior patterns and optimizes home appliances and energy usage based on predictions.

[1156] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[1157] A "smart home assistant" is a device that uses voice recognition and natural language processing to operate home appliances in response to user instructions.

[1158] "Automatic home appliance control" is a function that automatically operates home appliances based on user behavior patterns and predictions.

[1159] "Electricity usage monitoring" is a function that monitors electricity usage in homes and physical stores in real time and collects data.

[1160] "Voice or text electricity usage inquiry means" refers to an interface that allows a user to check electricity usage using voice commands or text input.

[1161] "Means for monitoring energy usage in physical stores in real time" is a system for monitoring energy consumption in physical stores in real time and collecting data.

[1162] "A means for store staff to inquire about energy usage by voice or text using a smartphone" is an interface that allows store staff to check energy usage by voice commands or text input using a smartphone.

[1163] The "means for providing energy saving advice based on energy usage data" is a function that analyzes collected energy usage data and provides advice to promote efficient energy usage.

[1164] As an embodiment of the present invention, a management system for energy usage in a physical store is constructed. The system includes generative AI, a smart home appliance control app, a smart home assistant, an electricity usage monitoring means, a means for inquiring about electricity usage by voice or text, a means for monitoring energy usage in the physical store in real time, a means for store staff to inquire about energy usage by voice or text using a smartphone, and a means for providing energy saving advice based on energy usage data.

[1165] System program generation

[1166] The system is built using a Python program, using the speech_recognition library for speech recognition and the pyttsx3 library for speech synthesis, allowing store staff to inquire about energy usage by voice or text using their smartphones.

[1167] Program processing explanation

[1168] The server receives voice input via a microphone and converts the speech to text using the speech_recognition library. It then queries the user for energy usage based on the converted text. It then obtains the corresponding energy usage from mock data and responds via voice using the pyttsx3 library.

[1169] Hardware and software used

[1170] Hardware: Smartphone (microphone, speaker)

[1171] Software: Python, speech_recognition library, pyttsx3 library

[1172] Specific examples

[1173] For example, if a store staff member asks into their smartphone, "How much electricity was used this month?", the application will respond, "This month's electricity usage is 350 kWh." If they ask, "How much electricity was used last month?", the application will respond, "Last month's electricity usage was 400 kWh."

[1174] Prompt Sentence Examples

[1175] "How much electricity did you use this month?"

[1176] Please tell me how much electricity you used last month.

[1177] In this way, efficient management of energy usage and power saving can be achieved even in physical stores.

[1178] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1179] Step 1:

[1180] The user launches the smartphone application and asks, "How much electricity have I used this month?"

[1181] Input: User's voice command

[1182] Output: Audio data

[1183] Step 2:

[1184] The terminal uses a microphone to capture the user's voice.

[1185] Input: Audio data

[1186] Output: Captured audio data

[1187] Step 3:

[1188] The device converts the acquired voice data into text using the speech_recognition library.

[1189] Input: Captured audio data

[1190] Output: Text data (e.g., "How much electricity did you use this month?")

[1191] Step 4:

[1192] The terminal analyzes the converted text data and recognizes that it is an inquiry about energy usage.

[1193] Input: Text data

[1194] Output: Analysis results (energy consumption query)

[1195] Step 5:

[1196] The server obtains the corresponding energy usage amount from the mock data based on the analysis results.

[1197] Input: Analysis results

[1198] Output: Energy usage data (e.g. 350 kWh)

[1199] Step 6:

[1200] The server converts the acquired energy usage data into a text response.

[1201] Input: Energy usage data

[1202] Output: Text response (e.g., "This month's electricity usage is 350 kWh.")

[1203] Step 7:

[1204] The device uses the pyttsx3 library to convert the text response into speech and transmits the response to the user through the speaker.

[1205] Input: Text response

[1206] Output: Voice data (e.g., "This month's electricity usage is 350 kWh.")

[1207] Step 8:

[1208] The user listens to the voice response from the terminal and confirms the amount of energy used.

[1209] Input: Audio data

[1210] Output: User understanding (checking energy usage)

[1211] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1212] "Example 1"

[1213] The generative AI works in conjunction with an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, behavioral patterns, etc. For example, if the user is speaking in a sad tone, the emotion engine conveys that information to the generative AI. The generative AI then automates home appliance control based on that information. For example, if the generative AI recognizes that the user is sad, it will perform controls such as dimming the lights in the room.

[1214] "Example 2"

[1215] The emotion engine changes the way it presents the electricity usage monitoring results depending on the user's emotions. For example, if the emotion engine recognizes that the user is angry, it will present the electricity usage monitoring results in a direct and concise manner. On the other hand, if the emotion engine recognizes that the user is relaxed, it will present the electricity usage monitoring results in a detailed and polite manner.

[1216] "Example 3"

[1217] Generative AI automates home appliance control based on information from the emotion engine. For example, if the generative AI recognizes that the user is happy, it can automatically activate a music player and play the user's favorite music. This makes it possible to provide a comfortable living space that responds to the user's emotions.

[1218] The processing flow of each embodiment will be described below.

[1219] "Example 1"

[1220] Step 1: The emotion engine recognizes emotions from the user's tone of voice, facial expressions, behavioral patterns, etc.

[1221] Step 2: The emotion engine communicates this information to the generative AI.

[1222] Step 3: The generative AI uses this information to automate home appliance control. For example, if the generative AI recognizes that the user is sad, it will dim the lights in the room.

[1223] "Example 2"

[1224] Step 1: The emotion engine recognizes the user's emotion.

[1225] Step 2: The emotion engine changes the way it presents the electricity usage monitoring results according to the user's emotions. For example, if it recognizes that the user is angry, the emotion engine will present the electricity usage monitoring results in a direct and concise manner.

[1226] "Example 3"

[1227] Step 1: The generative AI receives information from the emotion engine.

[1228] Step 2: The generative AI uses this information to automate home appliance control. For example, if the generative AI recognizes that the user is happy, it will automatically activate a music player and play the user's favorite music.

[1229] Example 1

[1230] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1231] Conventional smart home appliance control systems do not fully consider the user's behavioral patterns and emotions, making it difficult to achieve optimal home appliance control that suits the user's lifestyle. Furthermore, their ability to monitor electricity usage and offer energy-saving suggestions is limited, and improvements to improve user convenience are needed.

[1232] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1233] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for monitoring electricity usage, means for inquiring about electricity usage by voice or text, means for collecting user behavioral data, means for learning behavioral patterns, means for collecting emotion data, and means for recognizing emotions and linking home appliance control. This enables optimal home appliance control based on the user's behavioral patterns and emotions, and also enables detailed monitoring of electricity usage and energy-saving suggestions.

[1234] "Generative AI" is an artificial intelligence system that learns user behavior patterns and emotions and automates home appliance control based on that.

[1235] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[1236] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to comprehensively manage and control home appliances and systems.

[1237] "Automatic home appliance control" is a function that automatically controls the operation of home appliances based on the user's behavioral patterns and emotions.

[1238] "Electricity usage monitoring" is a function that monitors and records the power consumption of home appliances and systems in real time.

[1239] "Enquire about electricity usage by voice or text" is a function that allows users to check current electricity consumption status and past usage history using voice commands or text input.

[1240] "User behavior data" is data that indicates the behavioral patterns and habits of the user in their daily lives.

[1241] "Learning behavioral patterns" is the process of analyzing collected user behavior data and extracting specific patterns and trends.

[1242] "Emotion data" is data that indicates the emotional state of the user, which can be obtained from the user's tone of voice, facial expression, behavior, etc.

[1243] "Recognizing emotions and coordinating home appliance control" is a function that analyzes the user's emotional state and adjusts the operation of home appliances based on the results.

[1244] MODE FOR CARRYING OUT THE INVENTION

[1245] This invention is a system that realizes automatic home appliance control based on user behavior patterns and emotions by linking generative AI with a smart home appliance control app and a smart home assistant. Specific embodiments of this system are described below.

[1246] 1. Program Generation

[1247] The server learns user behavior patterns and generates a generative AI program that automates home appliance control based on the learning results. This program includes algorithms for collecting and analyzing user behavior data. Specifically, a machine learning model is built using software such as Python and TENSORFLOW (registered trademark).

[1248] 2. Program Processing

[1249] The server uses generative AI to learn the user's behavioral patterns. For example, if the server learns that the user wakes up at 7:00 every morning, it will open the curtains and turn on the coffee maker at that time. This process is performed using the API of the smart home device.

[1250] The generative AI also works in conjunction with the emotion engine. The emotion engine uses OpenCV and NLP (natural language processing) libraries to analyze emotions from the user's tone of voice and facial expressions. For example, if the user speaks in a sad tone, the emotion engine conveys that information to the generative AI. The generative AI then automates home appliance control based on that information. For example, if the generative AI recognizes that the user is sad, it will dim the lights in the room.

[1251] 3. Specific Examples

[1252] As a concrete example, consider a case where a user has the habit of waking up at 7am every morning and drinking coffee. The generative AI learns this pattern and opens the curtains and starts the coffee maker at 7am every morning. Also, if the user says "I'm tired today" in a sad tone of voice, the emotion engine relays this information to the generative AI, which then dims the lights in the room.

[1253] Prompt Sentence Examples

[1254] An example prompt is:

[1255] Explain how the generative AI would control the appliances if the user wakes up at 7am every morning and has the habit of drinking coffee. Also, explain how the generative AI would control the appliances if the user says "I'm tired today" in a sad tone of voice.

[1256] In this way, generative AI automates home appliance control based on user behavior patterns and emotions.

[1257] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1258] Program processing flow

[1259] Step 1: Collect user behavior data

[1260] The server collects user behavior data. Specifically, it acquires user activity data (e.g., heart rate, number of steps, wake-up time, etc.) through sensors in the smartwatch or smartphone.

[1261] Input: Sensor data from smartwatches and smartphones

[1262] Output: User behavior data (e.g., wake-up time, activity level)

[1263] Step 2: Learning behavioral patterns

[1264] The server uses the collected behavioral data to learn user behavior patterns. It uses Python and Tensorflow (registered trademark) to build a machine learning model and analyze user behavior patterns.

[1265] Input: User behavior data

[1266] Output: Learned behavioral patterns (e.g., waking up at 7am every morning)

[1267] Step 3: Automate home appliance control

[1268] The server automatically controls home appliances based on the learned behavioral patterns. It calls the API of the smart home device and operates the appliances according to the user's behavioral patterns. For example, if the server learns that the user wakes up at 7:00 every morning, it will open the curtains and start the coffee maker at that time.

[1269] Input: Learned behavioral patterns

[1270] Output: Home appliance control commands (e.g., open curtains, turn on coffee maker)

[1271] Step 4: Collecting emotion data

[1272] The device collects the user's emotional data, capturing their tone of voice and facial expressions through a smart speaker or camera, and sends the data to a server.

[1273] Input: Audio and video data from smart speakers and cameras

[1274] Output: User emotion data (e.g., tone of voice, facial expression)

[1275] Step 5: Coordination of emotion recognition and home appliance control

[1276] The server uses an emotion engine to recognize the user's emotions. It uses OpenCV and NLP libraries to analyze emotions from the user's tone of voice and facial expressions. Based on the analysis results, the generative AI automates home appliance control. For example, if it recognizes that the user is sad, it will dim the lights in the room.

[1277] Input: User emotion data

[1278] Output: Home appliance control command (e.g. dim the lights)

[1279] Step 6: Monitor your electricity usage

[1280] The server monitors and records the power consumption of home appliances and systems in real time, and provides detailed analysis of the power consumption of specific time periods and specific appliances to suggest ways to save energy.

[1281] Input: Power consumption data from appliances and systems

[1282] Output: Electricity usage report and energy saving suggestions

[1283] In this way, generative AI can automate home appliance control based on the user's behavioral patterns and emotions, while also providing detailed monitoring of electricity usage and energy-saving suggestions.

[1284] (Application example 1)

[1285] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1286] Conventional smart home appliance control systems can automate home appliances based on user behavior patterns, but they have the problem of not being able to adjust the environment based on the user's emotions. Furthermore, in brick-and-mortar stores, there is a need to optimize the store environment based on the emotions and behavior patterns of customers, but there was no system that could do this automatically. This made it difficult to improve customer satisfaction.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1288] In this invention, the server includes means for linking with the generative AI, the smart home appliance control app, and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for linking with an emotion engine that recognizes user emotions and adjusting the store environment, and means for automatically adjusting the lighting, music, and temperature of the store based on the user's behavioral patterns and emotions. This makes it possible to automatically optimize the store environment based on the emotions and behavioral patterns of customers.

[1289] "Generative AI" is artificial intelligence that learns user behavior patterns and emotions and automates the control of home appliances and the environment based on that.

[1290] A "smart home appliance control app" is an application for remotely controlling home appliances through devices such as smartphones and tablets.

[1291] A "smart home assistant" is a device that uses voice recognition and artificial intelligence to control home appliances.

[1292] "Automatic home appliance control" is a function that automatically operates home appliances based on the user's behavioral patterns and emotions.

[1293] "Electricity usage monitoring" is a function that monitors the amount of electricity consumed within a home or store in real time.

[1294] An "emotion engine" is software or hardware that recognizes emotions from a user's tone of voice, facial expressions, and behavioral patterns.

[1295] "Means for adjusting the store environment" refers to a system for controlling environmental elements such as lighting, music, and temperature within the store.

[1296] "Behavioral patterns" refer to the tendencies and habits of a user's daily behavior.

[1297] "Means for automatically adjusting lighting, music, and temperature" refers to a function that automatically adjusts lighting, music, and temperature in the store based on the user's behavioral patterns and emotions.

[1298] The system for implementing this invention includes a generative AI, a smart home appliance control app, a smart home assistant, an emotion engine, and a store environment adjustment means. The specific configuration and operation of the system are described below.

[1299] System configuration

[1300] 1. Generative AI

[1301] Generative AI is an artificial intelligence that learns user behavioral patterns and emotions and automates the control of home appliances and the environment based on that. For example, it is implemented using machine learning libraries such as Python and TENSORFLOW (registered trademark).

[1302] 2. Smart home appliance control app

[1303] A smart home appliance control app is an application that allows you to remotely control home appliances through devices such as smartphones and tablets. It runs on iOS and Android platforms.

[1304] 3. Smart Home Assistant

[1305] A smart home assistant is a device that uses voice recognition and artificial intelligence to control home appliances, such as Amazon Echo and Google Home.

[1306] 4. Emotion Engine

[1307] The emotion engine is software or hardware that recognizes emotions from the user's tone of voice, facial expressions, and behavioral patterns. For example, it uses OpenCV or Microsoft® Azure® emotion recognition APIs.

[1308] 5. Store environment adjustment measures

[1309] Store environment control means are systems for controlling environmental elements such as lighting, music, and temperature within a store, including smart lighting systems (such as Philips® Hue®) and smart thermostats (such as Nest).

[1310] System Operation

[1311] The server uses generative AI to learn the user's behavioral patterns and emotions. The emotion engine analyzes the user's tone of voice and facial expressions in real time and sends the results to the generative AI. The generative AI then uses this data to provide instructions for optimizing the store environment.

[1312] For example, when a customer walks into a store, a smartphone app captures the customer's tone of voice and facial expression. If the customer says in a sad tone, "I'm a little tired today," the app can use that information to dim the store's lights and change the music to something more soothing.

[1313] Specific examples

[1314] As a concrete example, the following prompt sentence is input into the generative AI model:

[1315] Prompt Sentence Examples

[1316] When a customer walks into a store, the smartphone app captures their tone of voice and facial expression. If the customer says in a sad tone, "I'm a little tired today," the app can use that information to dim the store's lights and change the music to something more soothing.

[1317] In this way, it becomes possible to automatically optimize the store environment based on the emotions and behavioral patterns of customers.

[1318] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1319] Step 1:

[1320] The user enters the store.

[1321] Input: User enters the store

[1322] Output: User's store entry information

[1323] Specific operation: The smartphone app detects when a user enters a store and sends the entry information to the server.

[1324] Step 2:

[1325] The device captures the user's tone of voice and facial expressions.

[1326] Input: User's tone of voice and facial expression

[1327] Output: Captured audio and image data

[1328] What it does: It uses the smartphone's microphone and camera to capture the user's tone of voice and facial expressions in real time.

[1329] Step 3:

[1330] The server sends the captured data to the emotion engine.

[1331] Input: Audio data and image data

[1332] Output: Sending data to the emotion engine

[1333] Specific operation: The smartphone app sends the captured audio and image data to the server, which then sends it to the emotion engine.

[1334] Step 4:

[1335] The emotion engine analyzes the user's emotions.

[1336] Input: Audio data and image data

[1337] Output: Parsed emotion data

[1338] Specific operation: The emotion engine analyzes voice and image data and recognizes the user's emotions (e.g., sad, happy, tired).

[1339] Step 5:

[1340] The server sends the analyzed emotion data to the generative AI.

[1341] Input: Parsed emotion data

[1342] Output: Sending data to generative AI

[1343] Specific operation: The emotion engine sends the analyzed emotion data to the server, and the server sends it to the generative AI.

[1344] Step 6:

[1345] Generative AI generates instructions to optimize the store environment based on user behavior patterns and emotions.

[1346] Input: Analyzed emotion data and behavior pattern data

[1347] Output: Store environment optimization instructions

[1348] Specific operation: Generative AI generates instructions to optimize the store's lighting, music, temperature, etc. based on the user's emotional data and past behavioral patterns.

[1349] Step 7:

[1350] The server transmits the generated instructions to the store environment adjustment means.

[1351] Input: Store environment optimization instructions

[1352] Output: Send instructions to store environment adjustment means

[1353] Specific operation: The server sends the instructions received from the generative AI to the store environment adjustment means.

[1354] Step 8:

[1355] The store environment adjusting means adjusts the lighting, music, and temperature of the store based on the instructions.

[1356] Input: Store environment optimization instructions

[1357] Output: Optimized store environment

[1358] Specific operation: Based on instructions from the server, the store environment adjustment means dims the lights, changes the music to a more calming one, and adjusts the temperature.

[1359] Example 2

[1360] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1361] Conventional smart home appliance control systems monitor electricity usage and offer energy-saving suggestions, but they have the problem of being unable to respond flexibly to user emotions. It is also difficult to analyze usage patterns for specific time periods or for each appliance in detail and offer efficient energy-saving suggestions.

[1362] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1363] In this invention, the server includes means for linking with the generative artificial intelligence, the smart home appliance control application, and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for recognizing the user's emotions and presenting the results of the electricity usage monitoring according to the emotions, and means for collecting electricity usage data and analyzing usage status for specific time periods and for each home appliance. This enables flexible responses according to the user's emotions and makes it possible to make efficient energy-saving suggestions by analyzing usage status for specific time periods and for each home appliance in detail.

[1364] "Generative AI" is an AI technology that learns user behavior patterns and automates home appliance control based on predictions.

[1365] A "smart home appliance control application" is software that allows remote control of home appliances using devices such as smartphones and tablets.

[1366] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to manage and control home appliances in an integrated manner.

[1367] "Automatic home appliance control" is a function that automatically operates home appliances based on user behavior patterns and predictions.

[1368] "Electricity usage monitoring" is the process of measuring the electricity usage of home appliances in real time and collecting and analyzing the data.

[1369] "Voice and text electricity usage inquiry means" refers to an interface that allows a user to obtain information about electricity usage using voice commands or text input.

[1370] "Means for recognizing user emotions" refers to technology that uses sensors such as cameras and microphones to analyze the user's facial expressions and tone of voice to determine their emotional state.

[1371] The "means for presenting the monitoring results of electricity usage according to emotions" is a function for displaying the monitoring results of electricity usage in different formats based on the emotional state of the user.

[1372] "Means of collecting electricity usage data" refers to technology that uses smart meters and IoT devices to measure the electricity usage of home appliances and send that data to a server.

[1373] "Means for analyzing usage patterns for specific time periods and for each home appliance" refers to the process of analyzing collected electricity usage data and extracting usage patterns for specific time periods and for each home appliance.

[1374] MODE FOR CARRYING OUT THE INVENTION

[1375] This invention is a system that links generative artificial intelligence with a smart home appliance control application and a smart home assistant, and has the function of presenting the results of monitoring electricity usage according to the user's emotions. Specific embodiments of this system are described below.

[1376] 1. System Overview

[1377] The system consists of three main components: a server, a terminal, and a user. The server collects and analyzes data, the terminal recognizes user emotions, and the user receives the results.

[1378] 2. Hardware and Software Used

[1379] Server: Database management system (e.g., MySQL (registered trademark)), data analysis tool (e.g., Python's Pandas library)

[1380] Device: Camera, microphone, emotion analysis software (e.g., Microsoft® Azure® Emotion API)

[1381] User: Devices such as smartphones and tablets

[1382] 3. Program Processing

[1383] The server collects real-time electricity usage data from each home appliance using smart meters and IoT devices. For example, sensors attached to air conditioners and microwave ovens measure electricity usage and send the data to the server.

[1384] The server stores the collected electricity usage data in a database and analyzes it using the Python Pandas library. Specifically, it extracts usage patterns for specific time periods and for each appliance, and identifies abnormal usage and opportunities for power saving. For example, if an air conditioner is used for more than eight hours a day, the data is flagged.

[1385] To recognize the user's emotions, the device uses a camera and microphone to capture the user's facial expressions and tone of voice. This uses the Microsoft® Azure® Emotion API. For example, when a user speaks in front of the device, the device analyzes their emotions from their tone of voice and facial expressions to determine whether they are angry or relaxed.

[1386] The server presents the results of electricity usage monitoring to the user based on the results of emotion recognition. If the user is angry, the results are presented briefly. For example, a message such as "You are using the air conditioner for too long. Please reduce it by one hour" is displayed. On the other hand, if the user is relaxed, the results are presented in detail. For example, a detailed suggestion such as "You are using the air conditioner for more than eight hours a day. By reducing this by one hour, you can save 500 yen on your monthly electricity bill" is displayed.

[1387] 4. Examples and prompts

[1388] Examples:

[1389] Air conditioner usage:

[1390] The server collects electricity usage data from air conditioners every hour and stores it in a database.

[1391] The server uses Python's Pandas library to flag any usage exceeding eight hours per day.

[1392] The device uses a camera and microphone to recognize emotions when the user is in front of the device.

[1393] If the server recognizes that the user is angry, it will simply say, "You've been using the air conditioner for too long. Please reduce it by one hour."

[1394] Frequency of microwave use:

[1395] The server tallyes the number of times the microwave is used each day and stores the count in a database.

[1396] The server uses Python's Pandas library to flag any usage exceeding five times in a day.

[1397] If the device recognizes that the user is relaxed, it will provide detailed information such as, "You use the microwave a lot. By using multiple devices at once, you can save on electricity bills."

[1398] Example prompt sentence:

[1399] "Analyze air conditioner usage time and provide energy-saving suggestions."

[1400] "When users are relaxed, provide detailed electricity usage monitoring results."

[1401] The system efficiently manages users' electricity usage and responds flexibly to their emotions, helping them live more comfortable and economical lives.

[1402] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1403] Step 1:

[1404] Collecting electricity usage data

[1405] The server collects electricity usage data from each home appliance in real time.

[1406] Input: Electricity usage data sent from smart meters and IoT devices.

[1407] Specific operation: Sensors attached to air conditioners and microwave ovens measure electricity usage and send the data to a server.

[1408] Output: Collected electricity usage data is stored in a database.

[1409] Step 2:

[1410] Data storage and initial processing

[1411] The server stores the collected electricity usage data in a database and performs initial processing.

[1412] Input: Electricity usage data collected in step 1.

[1413] Specific operations: Store data using a database management system (e.g., MySQL (registered trademark)) and clean and format the data using Python's Pandas library.

[1414] Output: Electricity usage data after initial processing.

[1415] Step 3:

[1416] Analyzing the data

[1417] The server analyzes the electricity usage data after the initial processing is completed.

[1418] Input: Electricity usage data after initial processing in step 2.

[1419] How it works: Using Python's Pandas library, it extracts usage patterns for specific time periods and for each appliance, and identifies abnormal usage and opportunities for power saving. For example, if an air conditioner is used for more than eight hours a day, it flags that data.

[1420] Output: Analysis of usage patterns and abnormal usage data.

[1421] Step 4:

[1422] emotion recognition

[1423] The device uses a camera and microphone to capture the user's facial expressions and tone of voice to recognize their emotions.

[1424] Input: The user's facial expression and tone of voice.

[1425] Specific operation: Analyzes the user's emotional state using emotion analysis software (e.g., Microsoft® Azure® Emotion API). For example, when a user speaks to the device, the device analyzes their emotions from their tone of voice and facial expressions to determine whether they are angry or relaxed.

[1426] Output: User's emotional state data.

[1427] Step 5:

[1428] Presentation of results

[1429] The server presents the user with the results of monitoring electricity usage based on the emotion recognition results.

[1430] Input: Analysis results from step 3 and emotional state data from step 4.

[1431] Specific behavior: If the user is angry, the results will be presented succinctly. For example, a message such as "You are using the air conditioner for too long. Please reduce it by one hour" will be displayed. On the other hand, if the user is relaxed, the results will be presented in detail. For example, a detailed suggestion such as "You are using the air conditioner for more than eight hours a day. By reducing this by one hour, you can save 500 yen on your monthly electricity bill" will be displayed.

[1432] Output: Electricity usage monitoring results and energy saving suggestions presented to the user.

[1433] (Application example 2)

[1434] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1435] Conventional electricity usage monitoring systems present information uniformly without considering the user's emotions, which means they are unable to provide appropriate information according to the user's situation and emotions. Furthermore, in physical stores, the lack of detailed analysis of electricity usage and energy-saving suggestions makes efficient energy management difficult.

[1436] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1437] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for detecting a user's emotions and presenting the results of electricity usage monitoring according to the emotions, means for monitoring electricity usage in a physical store in real time and performing detailed analysis of electricity usage during specific time periods and for specific devices, and means for presenting the results of electricity usage monitoring and making energy saving suggestions according to the emotions of the store manager. This makes it possible to provide appropriate information according to the user's emotions and enable efficient energy management in physical stores.

[1438] "Generative AI" is an artificial intelligence technology that generates new information and predictions based on data.

[1439] A "smart home appliance control app" is an application that allows you to remotely control home appliances using a smartphone or tablet.

[1440] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to comprehensively manage and control home appliances and systems.

[1441] "Automatic home appliance control" is a function that automatically operates home appliances based on user settings and behavior patterns.

[1442] "Electricity usage monitoring" is the process of monitoring and recording the electricity consumption of specific devices or at specific times in real time.

[1443] "Enquire about electricity usage by voice or text" is a function that allows the user to obtain information about electricity usage by voice instructions or text input.

[1444] "Detecting user emotions" refers to a technology that identifies a user's emotional state from their facial expressions, voice, behavior, etc.

[1445] "Presenting electricity usage monitoring results according to emotions" is a function that displays electricity usage information in an appropriate format based on the user's emotional state.

[1446] "Real-time monitoring of electricity usage in physical stores" refers to the process of monitoring electricity consumption in stores in real time and collecting data.

[1447] "Analyzing electricity usage in detail for specific time periods or specific devices" refers to the process of analyzing electricity consumption data in detail for specific time periods or specific devices to evaluate usage patterns and efficiency.

[1448] "Presenting the results of monitoring electricity usage and making energy-saving suggestions based on the store manager's emotions" is a function that displays electricity usage information in an appropriate format and makes suggestions to improve energy efficiency based on the store manager's emotional state.

[1449] A system for implementing this invention includes generative AI, a smart home appliance control app, a smart home assistant, an electricity usage monitoring means, a means for inquiring about electricity usage by voice or text, a means for detecting a user's emotions, a means for presenting the results of monitoring electricity usage according to the emotions, a means for monitoring electricity usage in a physical store in real time, a means for performing detailed analysis of electricity usage during specific time periods or for specific devices, and a means for presenting the results of monitoring electricity usage according to the emotions of the store manager and making energy saving suggestions.

[1450] Explanation of program processing

[1451] The server uses generative AI to learn user behavior patterns and automate home appliance control based on predictions. The smart home appliance control app and smart home assistant receive user voice instructions and text input and acquire electricity usage data. The electricity usage monitoring tool monitors power consumption during specific time periods and for specific devices in real time and collects data.

[1452] The means for detecting user emotions identifies the emotional state of the user from facial expressions, voice, behavior, etc. The means for presenting the monitoring results of electricity usage according to emotions displays the electricity usage information in an appropriate format based on the emotional state of the user. For example, if the user is angry, the information is presented in a concise manner, and if the user is relaxed, the information is presented in a detailed manner.

[1453] The method for monitoring electricity usage in a physical store in real time involves monitoring electricity consumption in the store in real time and collecting data. The method for analyzing electricity usage in detail during specific time periods or for specific devices involves analyzing electricity consumption data in detail during specific time periods or for specific devices to evaluate usage patterns and efficiency.

[1454] The means for presenting the monitoring results of electricity usage and making energy saving suggestions according to the store manager's emotions displays the electricity usage information in an appropriate format and makes suggestions for improving energy efficiency based on the store manager's emotional state.

[1455] Specific examples

[1456] For example, suppose a store manager opens an application on their smartphone. The application monitors the electricity usage of the air conditioner and microwave oven for 24 hours. If the manager's emotion is "angry," the application displays the total usage in a concise manner. If the manager's emotion is "relaxed," the application displays the detailed usage and peak times.

[1457] Example prompts to input to a generative AI model:

[1458] "Please develop an application that monitors electricity usage in physical stores and performs detailed analysis of electricity usage during specific time periods and for specific devices. In addition, please add a function that displays the results of electricity usage monitoring and makes energy-saving suggestions based on the store manager's emotions."

[1459] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1460] Step 1:

[1461] The server uses generative AI to learn user behavior patterns.

[1462] Input: User's past behavior data

[1463] Output: Model of behavioral patterns

[1464] How it works: The server collects data on the user's past behavior and inputs it into a generative AI model. The model learns the user's behavioral patterns and generates a predictive model.

[1465] Step 2:

[1466] The device receives user voice commands and text inputs using a smart home appliance control app and smart home assistant.

[1467] Input: User voice commands and text input

[1468] Output: Home appliance control commands

[1469] Specific operation: The device receives voice instructions or text input from the user, analyzes them, and generates home appliance control commands, which are then sent to the smart home appliances.

[1470] Step 3:

[1471] The server uses an electricity usage monitoring means to monitor the power consumption of a specific time period or a specific device in real time.

[1472] Input: Power consumption data

[1473] Output: Real-time power consumption report

[1474] Specific operation: The server collects power consumption data sent from each device and monitors it in real time. The collected data is compiled into a real-time power consumption report.

[1475] Step 4:

[1476] The terminal uses a means for detecting the user's emotions to identify the user's emotional state from facial expressions, voice, actions, etc.

[1477] Input: User facial, voice, and behavioral data

[1478] Output: Emotional state

[1479] How it works: The device collects the user's facial, voice, and behavioral data through the camera and microphone, and inputs it into an emotion detection algorithm. The algorithm identifies the user's emotional state and outputs the results.

[1480] Step 5:

[1481] The server uses the means for presenting the electricity usage monitoring results according to emotions to display the information in an appropriate format based on the emotional state of the user.

[1482] Input: Emotional state, power consumption data

[1483] Output: Emotion-based power consumption report

[1484] Specific operation: The server combines the user's emotional state with the power consumption data to generate a power consumption report according to the user's emotions. For example, if the user is angry, it generates a concise report, and if the user is relaxed, it generates a detailed report.

[1485] Step 6:

[1486] The server monitors the electricity usage of the physical store in real time and performs detailed analysis of electricity usage during specific time periods and for specific devices.

[1487] Input: In-store electricity consumption data

[1488] Output: Detailed power consumption analysis report

[1489] Specific operation: The server collects power consumption data sent from each device in the store and analyzes it in detail, focusing on specific time periods or specific devices. The analysis results are compiled into a detailed power consumption analysis report.

[1490] Step 7:

[1491] The server presents the results of monitoring electricity usage and makes energy saving suggestions according to the store manager's emotions.

[1492] Input: Manager's emotional state, power consumption data

[1493] Output: Energy saving suggestion report according to emotions

[1494] Specific operation: The server combines the manager's emotional state with power consumption data to generate a report of energy-saving suggestions according to the manager's emotions. For example, if the manager is stressed, it will provide brief suggestions, and if the manager is relaxed, it will provide detailed suggestions.

[1495] Example 3

[1496] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1497] In conventional smart home systems, users have limited means to easily check their electricity usage, and home appliances are not controlled according to the user's emotions, making it difficult to provide a comfortable living space. Furthermore, there has been a lack of systems that take into account the user's behavioral patterns and emotions when it comes to monitoring electricity usage and automating home appliance control.

[1498] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1499] In this invention, the server includes means for linking the generative AI with the smart home appliance control application and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for generating instructions for controlling home appliances based on information from the emotion engine, and means for controlling home appliances according to the user's emotions. This allows the user to easily check electricity usage and enjoy a comfortable living space according to their emotions.

[1500] "Generative AI" is artificial intelligence that analyzes users' behavioral patterns and emotions and generates instructions for controlling home appliances based on that.

[1501] A "smart home appliance control application" is software for controlling home appliances in a smart home.

[1502] A "smart home assistant" is a device that receives user instructions via voice or text and operates home appliances and systems accordingly.

[1503] "Automatic home appliance control" is a function that automatically operates home appliances based on the user's behavioral patterns and emotions.

[1504] "Electricity usage monitoring" is a function that monitors and records the amount of electricity consumed within the home in real time.

[1505] The "emotion engine" is a system that analyzes the user's emotions from facial expressions, tone of voice, etc. and provides that information.

[1506] "Instructions for controlling home appliances" are commands for operating home appliances that are generated by generative AI based on the user's emotions and behavioral patterns.

[1507] The "means for controlling home appliances according to the user's emotions" is a function that operates home appliances according to the user's emotions based on information from the emotion engine.

[1508] This invention is a system that allows users to inquire about electricity usage by voice or text, and automates home appliance control based on information from a generative AI emotion engine. A specific embodiment of this system is described below.

[1509] System configuration

[1510] The system consists of the following major components:

[1511] 1. Smart home assistant: A device that receives voice and text queries from the user.

[1512] 2. Server: The central system for data processing and analysis.

[1513] 3. Generative AI: Artificial intelligence that analyzes user behavior patterns and emotions and generates instructions for controlling home appliances.

[1514] 4. Emotion engine: A system that analyzes emotions from the user's facial expressions and tone of voice.

[1515] 5. Smart Meter: A device that monitors electricity usage in the home in real time.

[1516] 6. Music player: A device that plays music based on instructions from generative AI.

[1517] Program processing

[1518] Electricity usage inquiries

[1519] 1. A user asks their smart home assistant via voice or text, "How much electricity have I used this month?"

[1520] 2. The device (smart home assistant) receives this query and sends a data request to the server.

[1521] 3. The server collects electricity usage data from the smart meter, analyzes it, and sends it to the device.

[1522] 4. The device responds to the user, "This month's electricity usage is 200 kWh."

[1523] Automated home appliance control

[1524] 1. The emotion engine analyzes the user's facial expressions and tone of voice and recognizes that the user is happy.

[1525] 2. The server receives the user's emotion information from the emotion engine.

[1526] 3. The server requests the generative AI to issue instructions for controlling home appliances based on the user's emotions.

[1527] 4. The generative AI generates instructions to control home appliances based on the user's emotions. For example, if the user is happy, it generates an instruction to a music player to "play the user's favorite music."

[1528] 5. The server sends instructions from the generative AI to the music playback device.

[1529] 6. The music player receives the command and plays the music.

[1530] Specific examples

[1531] A user asks their smart home assistant, "How much electricity have I used this month?"

[1532] The terminal receives this inquiry and sends a data request to the server.

[1533] The server obtains the data "200 kWh" from the smart meter, analyzes it, and sends it to the terminal.

[1534] The device responds to the user, "This month's electricity usage is 200 kWh."

[1535] The emotion engine recognizes that the user is happy based on their facial expression and tone of voice.

[1536] The server requests the generative AI to issue instructions to the music playback device.

[1537] The generative AI generates an instruction such as "Play the user's favorite music," and the server sends it to the music playback device.

[1538] The music player receives the instruction and plays the music.

[1539] Prompt Sentence Examples

[1540] "Tell me how much electricity I used this month."

[1541] "Set music to play when the user is happy"

[1542] This system allows the user to easily check the amount of electricity used and enjoy a comfortable living space according to their feelings. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1543] Electricity usage inquiries

[1544] Step 1:

[1545] A user asks their smart home assistant via voice or text, "How much electricity did I use this month?"

[1546] Input: User's voice or text query

[1547] Output: Query request to smart home assistant

[1548] Step 2:

[1549] The device (smart home assistant) receives this query and sends a data request to the server.

[1550] Input: User inquiry request

[1551] Data processing: Analyzes the query and generates a data request to send to the server

[1552] Output: Data request to the server

[1553] Step 3:

[1554] The server obtains electricity usage data from the smart meter.

[1555] Input: Data request from terminal

[1556] Data processing: Obtaining electricity usage data from smart meters

[1557] Output: Acquired electricity usage data

[1558] Step 4:

[1559] The server analyzes the acquired data and generates information to provide to the user.

[1560] Input: Acquired electricity usage data

[1561] Data calculation: Analysis of electricity usage data

[1562] Output: Analysis results (e.g. 200 kWh)

[1563] Step 5:

[1564] The server sends the analysis results to the device.

[1565] Input: Analysis results

[1566] Data processing: Converting analysis results into a data format for sending to the terminal

[1567] Output: Sending analysis results to the device

[1568] Step 6:

[1569] The device responds to the user by voice or text, saying, "This month's electricity usage is 200 kWh."

[1570] Input: Analysis results from the server

[1571] Data processing: Converting analysis results into audio or text to communicate to users

[1572] Output: Answer to the user

[1573] Automated home appliance control

[1574] Step 1:

[1575] The emotion engine analyzes the user's facial expressions and tone of voice and recognizes that the user is happy.

[1576] Input: User facial expressions and tone of voice

[1577] Data Computing: Sentiment Analysis

[1578] Output: User's emotional state (e.g., happy)

[1579] Step 2:

[1580] The server receives the user's emotion information from the emotion engine.

[1581] Input: Emotion information from the emotion engine

[1582] Output: Receiving emotional information

[1583] Step 3:

[1584] The server requests the generative AI to issue instructions for controlling home appliances based on the user's emotions.

[1585] Input: Emotion information

[1586] Data processing: Generate requests to generate instructions for controlling home appliances based on emotional information

[1587] Output: Requirements for generative AI

[1588] Step 4:

[1589] Generative AI generates instructions for controlling home appliances based on the user's emotions.

[1590] Input: Request from server

[1591] Data calculation: Generating instructions for controlling home appliances based on emotional information

[1592] Output: Instructions for controlling home appliances (e.g., instructing a music player to "play the user's favorite music")

[1593] Step 5:

[1594] The server sends instructions from the generative AI to the music playback device.

[1595] Input: Instructions for controlling home appliances from generative AI

[1596] Data processing: Converting instructions into a data format for sending to a music playback device

[1597] Output: Sending instructions to a music player

[1598] Step 6:

[1599] The music player receives the instruction and plays the music.

[1600] Input: Instructions from the server

[1601] Data processing: Play music based on instructions

[1602] Output: Music playback

[1603] In this way, the user can easily check the amount of electricity used and can enjoy a comfortable living space according to their feelings.

[1604] (Application example 3)

[1605] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1606] Conventional self-driving vehicles do not adjust the in-car environment according to passengers' emotions, and there is a lack of personalized services to make passengers comfortable. Furthermore, there are limited functions for monitoring and querying electricity usage, making it difficult for passengers to understand the vehicle's status in real time. This leads to lower passenger satisfaction and makes it difficult to provide a comfortable travel experience.

[1607] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1608] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for monitoring electricity usage, means for inquiring about electricity usage by voice or text, means for recognizing user emotions using an emotion recognition engine, and means for automatically adjusting the in-car environment according to the user's emotions. This makes it possible to provide a personalized in-car environment according to the passenger's emotions, resulting in a comfortable travel experience.

[1609] "Generative AI" is artificial intelligence that analyzes a user's emotions and behavioral patterns and provides appropriate responses and control based on that.

[1610] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[1611] A "smart home assistant" is a system that receives user instructions via voice or text and controls home appliances and other smart devices.

[1612] "Automatic home appliance control" is a function that automatically operates home appliances based on user instructions and behavioral patterns.

[1613] "Electricity usage monitoring" is a function that monitors the electricity usage of specific time periods or specific home appliances in real time, and collects and analyzes the data.

[1614] "Means to inquire about electricity usage by voice or text" is a function that allows users to inquire about information about electricity usage by voice or text.

[1615] An "emotion recognition engine" is a system that analyzes the user's emotions from their facial expressions and voice, and uses that information to provide appropriate responses and control.

[1616] "Means for automatically adjusting the in-car environment according to the user's emotions" is a function that automatically adjusts environmental settings such as lighting, music, and temperature in the car based on information obtained from the emotion recognition engine.

[1617] As an embodiment of the present invention, a system for recognizing the emotions of passengers in an autonomous vehicle and automatically adjusting the in-vehicle environment accordingly is provided. Specific embodiments of this system are described below.

[1618] System configuration

[1619] This system is configured using the following hardware and software.

[1620] 1. Hardware:

[1621] Camera: Used to capture passengers' facial expressions.

[1622] Microphone: Used to capture passenger voices.

[1623] Speaker: Used for voice response and music playback.

[1624] Display: Used to display text information.

[1625] 2. Software:

[1626] Emotion recognition engine: Used to recognize emotions from facial expressions and voice (e.g., emotion recognition service).

[1627] Generative AI: Used to adjust the in-car environment based on information from emotion recognition engines (e.g., generative AI models).

[1628] Smart home assistants: Used to respond to voice and text inquiries (e.g., voice assistants).

[1629] Data processing and calculation

[1630] 1. Emotion recognition:

[1631] Data obtained from the camera and microphone is input into an emotion recognition engine to analyze passenger emotions.

[1632] The emotion recognition engine analyzes facial expressions and voice characteristics to recognize passengers' emotions, such as "happy" or "relaxed."

[1633] 2. Generative AI prompt generation:

[1634] Based on the emotional information obtained from the emotion recognition engine, the generative AI generates appropriate prompts.

[1635] For example, if the passenger is happy, generate a prompt saying "If the passenger is happy, play their favorite music."

[1636] 3. Smart home assistant response:

[1637] Based on the generated prompts, the smart home assistant generates a response via voice or text.

[1638] For example, in response to the query, "How much electricity did you use this month?", the smart home assistant will respond, "Your electricity usage this month is 100kWh."

[1639] Specific examples

[1640] 1. Examples of emotion recognition:

[1641] The camera captures passengers' smiles and the microphone captures their laughter.

[1642] An emotion recognition engine analyzes this data and determines that the passenger is "happy."

[1643] 2. Examples of prompts:

[1644] "If passengers are happy, play their favorite music."

[1645] "If passengers want to relax, adjust the lighting to a warmer color."

[1646] This system makes it possible to provide a personalized in-car environment that responds to passenger emotions, resulting in a comfortable travel experience.

[1647] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1648] Step 1:

[1649] Capturing Emotional Data

[1650] The server captures passengers' facial expressions and voices using a camera and microphone. The video data from the camera and the audio data from the microphone are input. These data are sent to an emotion recognition engine.

[1651] Step 2:

[1652] emotion recognition

[1653] The server analyzes the captured facial and audio data using an emotion recognition engine. It extracts facial features and tone of voice from the input video and audio data to recognize the passenger's emotions. For example, if a smile or laughter is detected, it is recognized as "happy." The recognized emotional information is generated as output.

[1654] Step 3:

[1655] Prompt Generation

[1656] The server uses generative AI to generate prompts based on the emotional information obtained from the emotion recognition engine. Based on the input emotional information, it generates a prompt sentence containing an appropriate response or control instruction. For example, a generated prompt might say, "If the passenger is happy, play their favorite music." The generated prompt sentence is obtained as the output.

[1657] Step 4:

[1658] Implementing environmental adjustment instructions

[1659] The server sends the generated prompt sentence to the smart home assistant. The smart home assistant executes an instruction to adjust the in-car environment based on the prompt sentence. For example, it sends an instruction to a music player to "play your favorite music." As an output, the in-car environment is adjusted.

[1660] Step 5:

[1661] Voice and text response

[1662] When a user inquires about electricity usage by voice or text, the server receives the inquiry through the smart home assistant. The server analyzes the input voice or text data to obtain information about electricity usage. For example, in response to the inquiry "How much electricity was used this month?", the server generates a response such as "This month's electricity usage is 100kWh." The response is obtained as voice or text as output.

[1663] In this way, through a series of processing steps, the server provides a personalized in-car environment according to the passenger's emotions and responds to inquiries about electricity usage.

[1664] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1665] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / OpenAI.com / blog / ChatGPT> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1666] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1667] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1668] [Third embodiment]

[1669] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1670] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1671] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1672] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1673] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1674] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1675] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1676] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1677] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1678] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1679] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1680] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1681] "Example 1"

[1682] In one embodiment of the present invention, generative AI learns user behavior patterns and automates home appliance control based on the results of that learning. For example, if it learns that the user wakes up at 7:00 every morning, it will open the curtains and turn on the coffee maker at that time.

[1683] "Example 2"

[1684] In one embodiment of the present invention, the electricity usage monitoring means analyzes in detail the electricity usage of specific time periods or specific home appliances and makes energy saving suggestions. For example, the means analyzes the time periods when an air conditioner is running or when a microwave oven is used, and presents suggestions to the user on optimizing the usage time and frequency of those home appliances.

[1685] "Example 3"

[1686] Furthermore, as one embodiment of the present invention, a means for inquiring about electricity usage by voice or text is provided. A user can inquire of the smart home assistant, such as "How much electricity have I used this month?", and the smart home assistant will respond to the inquiry based on the electricity usage information obtained from the monitoring means.

[1687] The processing flow of each embodiment will be described below.

[1688] "Example 1"

[1689] Step 1: The generative AI learns the user's behavioral patterns. For example, it learns that the user wakes up at 7am every morning.

[1690] Step 2: Based on the learned behavioral patterns, the generative AI automates home appliance control, for example, automatically opening the curtains at 7:00 AM, the time when the user wakes up.

[1691] Step 3: Similarly, it also controls the coffee maker to start automatically, so that the user can enjoy a cup of coffee as soon as they wake up.

[1692] "Example 2"

[1693] Step 1: The electricity usage monitoring method analyzes in detail the electricity usage of specific time periods or specific home appliances, for example, analyzing the time periods when the air conditioner is running.

[1694] Step 2: Based on the analysis results, the electricity usage monitoring tool will make energy-saving suggestions to the user, such as optimizing the usage time and frequency of the air conditioner.

[1695] "Example 3"

[1696] Step 1: The user makes a voice or text inquiry to the smart home assistant about their electricity usage, such as "How much electricity did I use this month?"

[1697] Step 2: The smart home assistant responds to the query based on the electricity usage information obtained from the monitoring method, allowing the user to easily understand their electricity usage.

[1698] Example 1

[1699] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1700] Conventional smart home systems require users to manually operate home appliances, limiting the scope of automation. Furthermore, there is a lack of monitoring of electricity usage and energy-saving suggestions, making it difficult to improve energy efficiency. Furthermore, there is no way to control home appliances based on user behavior patterns, resulting in low user convenience.

[1701] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1702] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for monitoring electricity usage, means for inquiring about electricity usage by voice or text, means for collecting user behavioral data, means for analyzing and learning from the behavioral data, means for generating a home appliance control schedule based on the learning results, and means for controlling home appliances based on the home appliance control schedule. This enables automatic home appliance control based on the user's behavioral patterns, thereby improving energy efficiency and user convenience.

[1703] "Generative AI" is an artificial intelligence technology that learns user behavior patterns and automates home appliance control based on the results of that learning.

[1704] A "smart home appliance control app" is application software that allows you to remotely control and manage home appliances using devices such as smartphones and tablets.

[1705] A "smart home assistant" is a digital assistant that uses voice recognition and natural language processing to operate home appliances according to user instructions.

[1706] "Automatic home appliance control" is a function that automatically operates home appliances based on the user's behavioral patterns and schedule.

[1707] "Electricity usage monitoring" is a function that monitors and records the power consumption of home appliances and the entire system in real time.

[1708] The "means for inquiring about electricity usage by voice or text" is an interface that allows users to check current electricity consumption status and past usage history using voice commands or text input.

[1709] "Means for collecting user behavioral data" refers to a function that uses smart home devices and sensors to record user behavior and environmental information.

[1710] "Means for analyzing and learning from behavioral data" refers to algorithms and artificial intelligence technologies that analyze collected user behavioral data and find patterns and regularities.

[1711] The "means for generating a home appliance control schedule based on the learning results" is a function that determines the optimal timing and method for operating home appliances based on the analyzed and learned user behavior patterns.

[1712] The "means for controlling home appliances based on the home appliance control schedule" refers to an interface or control system for actually operating home appliances according to the generated schedule.

[1713] MODE FOR CARRYING OUT THE INVENTION

[1714] This invention relates to a system that uses generative AI to learn user behavior patterns and automates home appliance control based on the learning results. Specific embodiments of this system are described below.

[1715] 1. Program Generation

[1716] The server uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate a program that learns user behavior patterns. This program has the function of collecting and analyzing user behavior data and generating a home appliance control schedule based on the learning results.

[1717] 2. Program Processing

[1718] The server collects user behavior data from smart home devices (e.g., smart speakers, smart lights, smart plugs, etc.), including the time and frequency with which users interact with the devices and data from environmental sensors (e.g., temperature, light intensity, motion detection, etc.).

[1719] The server then inputs the collected behavioral data into a generative AI model, which analyzes and learns the user's behavioral patterns. The AI ​​model then finds patterns in this data and identifies the user's typical behavioral schedule.

[1720] The server then generates a schedule for controlling home appliances based on the learning results. For example, the server might send an instruction to the smart curtain device to open the curtains at 7:00 every morning, and an instruction to the smart light device to turn off the lights at 10:00 every night.

[1721] Finally, the terminal (smart home device) receives instructions from the server and actually controls the home appliances, thereby realizing automation that matches the user's behavioral patterns.

[1722] 3. Specific Examples

[1723] As a concrete example, consider a case where a user has the habit of going to bed at 10 p.m. every night. In this case, the server will set the system to automatically turn off the lights and switch the air conditioner to energy-saving mode at 10 p.m.

[1724] Example prompt sentence:

[1725] Design a system that learns user behavior patterns and automates home appliance control. For example, if the user wakes up at 7:00 every morning, design the system to open the curtains and turn on the coffee maker at that time.

[1726] In this way, the server utilizes generative AI models to realize automated home appliance control to make users' lives more comfortable.

[1727] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1728] Program processing flow

[1729] Step 1: Collect user behavior data

[1730] The server collects user behavior data from smart home devices (e.g., smart speakers, smart lights, smart plugs, etc.), such as the time and frequency at which users operate the devices, and data from environmental sensors (e.g., temperature, illuminance, motion detection, etc.).

[1731] Input: Operation logs from smart home devices, environmental sensor data

[1732] Data processing: Organize and save operation logs and sensor data as time-series data

[1733] Output: Organized behavioral data

[1734] Specific behavior:

[1735] The server records the time when the user says "good morning" from the smart speaker.

[1736] The server records the on / off times of the smart lights.

[1737] Step 2: Analyze and learn from behavioral data

[1738] The server inputs the collected behavioral data into a generative AI model (e.g., GPT-4 (registered trademark)) to analyze and learn the user's behavioral patterns. The AI ​​model finds regularities and patterns in this data and identifies the user's typical behavioral schedule.

[1739] Input: Organized behavioral data

[1740] Data Computation: Using generative AI models to analyze behavioral data and learn patterns

[1741] Output: User behavior patterns

[1742] Specific behavior:

[1743] The server learns the pattern of users saying "good morning" every morning at 7am.

[1744] The server learns the pattern that the smart lights turn off at 10pm every night.

[1745] Step 3: Generate a home appliance control schedule

[1746] The server generates a schedule for controlling home appliances based on the learning results, which includes instructions for operating specific devices at specific times.

[1747] Input: User behavior patterns

[1748] Data processing: Determine the optimal timing and method for operating home appliances based on behavioral patterns

[1749] Output: Home appliance control schedule

[1750] Specific behavior:

[1751] The server sends an instruction to the smart curtain device to open the curtains at 7am every morning.

[1752] The server sends an instruction to the smart lights to turn off the lights at 10pm every night.

[1753] Step 4: Implementing appliance control

[1754] The terminals (smart home devices) receive instructions from the server and actually control the home appliances, thereby realizing automation that matches the user's behavioral patterns.

[1755] Input: Home appliance control schedule

[1756] Data calculation: Operate home appliances according to a schedule

[1757] Output: Actual operation of controlled appliances

[1758] Specific behavior:

[1759] The device receives instructions from the server and opens the curtains at 7:00 every morning.

[1760] The terminal receives instructions from the server to turn off the lights at 10 p.m. every night.

[1761] (Application example 1)

[1762] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1763] Conventional smart home appliance control systems are specialized for controlling home appliances in the home, and do not support automatic control of equipment in commercial facilities such as brick-and-mortar stores. Furthermore, manually controlling in-store equipment places a heavy burden on store staff, hindering efficient store operations. Furthermore, improper control of in-store equipment can result in energy waste and reduced customer satisfaction. To solve these issues, a system that can learn user behavior patterns and automatically control equipment is needed, even in brick-and-mortar stores.

[1764] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1765] In this invention, the server includes means for linking the generative AI with the smart home appliance control app and the smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for learning user behavior patterns and automatically controlling in-store equipment, and means for controlling in-store equipment. This enables automatic control of equipment based on user behavior patterns even in physical stores, reducing the burden on store staff and enabling efficient store management.

[1766] "Generative AI" is a type of artificial intelligence that learns user behavior patterns and automatically controls itself based on the results of that learning.

[1767] A "smart home appliance control app" is an application that runs on mobile devices such as smartphones and tablets and allows remote control of home appliances.

[1768] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to control home appliances and facilities.

[1769] "Automatic home appliance control" is a function that automatically operates home appliances based on user behavior patterns and predictions.

[1770] "Electricity usage monitoring" is a function that monitors and records the electricity consumption of specific time periods or specific home appliances in real time.

[1771] "A means for querying electricity usage via voice or text" refers to an interface that allows users to check their current electricity consumption status using voice commands or text input.

[1772] "Means of learning user behavior patterns and automatically controlling in-store equipment" refers to a function that analyzes the behavior of store staff and customers and automatically operates store equipment such as lighting, music, and air conditioning based on the results.

[1773] "Means for controlling in-store equipment" refers to systems or devices for manually or automatically operating in-store equipment such as lighting, music, and air conditioning.

[1774] A system for implementing this invention includes generative AI, a smart home appliance control app, a smart home assistant, a means for monitoring electricity usage, a means for inquiring about electricity usage via voice or text, a means for learning user behavior patterns and automatically controlling in-store equipment, and a means for controlling in-store equipment.

[1775] System Program

[1776] The system's program is implemented using programming languages ​​such as Python. The generative AI learns user behavior patterns and automatically controls in-store equipment based on the results of that learning. Specifically, it performs the following processes:

[1777] Processing Description

[1778] 1. Generative AI learning phase

[1779] The server collects and analyzes past behavioral data to learn user behavior patterns, including when store staff open the store and when they operate specific equipment, and uses machine learning algorithms to model these patterns.

[1780] 2. Predicting behavioral patterns

[1781] The server predicts future behavior based on learned behavioral patterns. For example, if it learns that store staff open the store at 9:00 a.m. every morning, it will automatically control the store's equipment to match that time.

[1782] 3. Automatic control of equipment

[1783] Based on predicted behavioral patterns, the server automatically controls in-store equipment, such as turning on lights, playing music, and setting the air conditioning to an appropriate temperature.

[1784] 4. Monitoring electricity usage

[1785] The server monitors and records electricity usage during specific time periods and for specific equipment in real time, preventing energy waste and enabling efficient energy management.

[1786] 5. Voice and text inquiries

[1787] Users can check the current power consumption status using voice commands or text input, allowing them to understand power consumption in real time and operate the equipment as needed.

[1788] Hardware and software used

[1789] Hardware: Servers, smartphones, tablets, smart home assistant devices

[1790] Software: Python, machine learning libraries (e.g., TENSORFLOW (registered trademark), scikit-learn (registered trademark)), smart home appliance control app

[1791] Specific examples

[1792] For example, by inputting the following prompt sentence into a generative AI model, it can learn the behavioral patterns of store clerks.

[1793] Learn the behavioral patterns of store staff who open the store at 9am every morning.

[1794] By using this prompt, the generative AI model can learn the behavioral patterns of store staff and automatically control in-store equipment based on the results, thereby reducing the burden on store staff and enabling efficient store operations.

[1795] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1796] Step 1:

[1797] The server collects user behavior data. Specifically, it obtains data such as when store staff open the store and when specific equipment is operated from smart home appliance control apps and smart home assistants. The input is past behavior data, and the output is a list of collected behavior data.

[1798] Step 2:

[1799] The server analyzes the collected behavioral data and learns the user's behavioral patterns. Specifically, it uses a machine learning algorithm to model the behavioral patterns. The input is a list of collected behavioral data, and the output is a learned behavioral pattern model.

[1800] Step 3:

[1801] The server predicts future behavior based on the learned behavioral patterns. Specifically, it uses a behavioral pattern model to predict what behavior will occur at a specific time. The input is the learned behavioral pattern model, and the output is a list of predicted behaviors.

[1802] Step 4:

[1803] The server automatically controls the equipment in the store based on the predicted behavior. Specifically, it performs operations such as turning on the lights, playing music, and setting the air conditioning to an appropriate temperature. The input is a list of predicted behaviors, and the output is the controlled status of the equipment.

[1804] Step 5:

[1805] The server monitors and records the electricity usage of specific time periods and specific equipment in real time. Specifically, it obtains power consumption data from smart home appliance control apps and smart home assistants and stores it in a database. The input is the power consumption data, and the output is the recorded power consumption data.

[1806] Step 6:

[1807] Users can check the current power consumption status using voice commands or text input. Specifically, they query the server through a smart home appliance control app or smart home assistant. The input is a voice command or text input, and the output is information about the current power consumption status.

[1808] Example 2

[1809] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1810] Conventional household electricity usage monitoring systems do not adequately collect and analyze electricity usage data, making it difficult to provide specific energy-saving suggestions to users. They also lack the functionality to learn user behavior patterns and automate home appliance control based on predictions. This has led to the issue of being unable to achieve efficient power management and energy saving.

[1811] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for linking with the generative artificial intelligence, the home appliance control application, and the home assistant, a means for realizing automatic home appliance control, a means for realizing electricity usage monitoring, a means for inquiring about electricity usage by voice or text, a means for collecting electricity usage data, a means for storing the collected data, a means for analyzing the stored data, a means for generating power-saving suggestions based on the analysis results, and a means for presenting the generated power-saving suggestions to the user. This enables detailed monitoring and analysis of electricity usage and provides specific power-saving suggestions to the user. Furthermore, efficient power management and power saving can be achieved by learning user behavior patterns and automating home appliance control based on predictions.

[1812] "Generative AI" is an AI technology that learns user behavior patterns and generates home appliance control and energy-saving suggestions based on predictions.

[1813] A "home appliance control application" is software for controlling home appliances through devices such as smartphones and tablets.

[1814] A "home assistant" is a device or software that interacts with users through voice or text to control home appliances and provide information.

[1815] "Automatic home appliance control" is a function that automatically adjusts the on / off and settings of home appliances based on user behavior patterns and predictions.

[1816] "Electricity usage monitoring" is the process of recording the electricity usage of each household appliance in real time and collecting it as data.

[1817] "A means of inquiring about electricity usage via voice or text" refers to an interface that allows users to check current electricity usage and past usage history through voice commands or text input.

[1818] The "means of collecting electricity usage data" refers to a system that uses smart meters and IoT devices to obtain real-time electricity usage data for each household appliance.

[1819] "Means for storing collected data" refers to a function that stores collected electricity usage data in a database so that it can be analyzed and referenced later.

[1820] "Means for analyzing stored data" refers to software or algorithms for analyzing stored electricity usage data and detecting usage patterns or anomalies.

[1821] The "means for generating energy-saving suggestions based on the analysis results" refers to a generative artificial intelligence that suggests specific energy-saving methods to users based on the results of data analysis.

[1822] The "means for presenting the generated power saving suggestions to the user" is an interface that notifies the user's terminal of the generated power saving suggestions and presents them visually or audibly.

[1823] The present invention is a system for monitoring the amount of electricity used in a home in detail and providing specific energy-saving suggestions to the user. A specific embodiment of this system will be described below.

[1824] The server first collects real-time electricity usage data from each household appliance using smart meters and IoT devices. These devices record the usage time and power consumption of appliances such as air conditioners and microwave ovens. For example, if the air conditioner is running, it records that the power consumption is 1.5 kWh per hour.

[1825] The server then stores the collected electricity usage data in a database, which contains detailed information about each appliance, such as usage time, frequency, and power consumption. For example, the data might be recorded in the format "October 1, 2023, 10:00 - Air conditioner - 1.5kWh."

[1826] The server analyzes the stored data using data analysis software such as Python and R. Specifically, it extracts electricity usage for specific time periods and for specific home appliances and analyzes usage patterns. For example, it confirms that air conditioner usage is concentrated between 9:00 AM and 5:00 PM.

[1827] The server then uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate power-saving suggestions based on the results of the data analysis. The following prompt is input to the generative AI model: "Generate power-saving suggestions for the user based on the following data. Data: Air conditioner usage time: 9:00 AM to 5:00 PM, Usage frequency: 5 times a day. Suggestion: How to optimize air conditioner usage time."

[1828] The generative AI model generates a suggestion such as, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM. We especially recommend refraining from using it in the morning when the outside temperature is relatively low."

[1829] Finally, the server sends the generated energy-saving suggestions to the user's device. The user can check the suggestions via their smartphone or computer. For example, a notification will be displayed saying, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM."

[1830] In this way, the server can monitor and analyze electricity usage in detail and provide specific energy-saving suggestions to users. It can also learn user behavior patterns and automate home appliance control based on predictions, achieving efficient power management and energy savings.

[1831] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1832] Step 1: Data collection

[1833] The server collects real-time electricity usage data from each home appliance via smart meters and IoT devices. The server receives data on the usage time and power consumption of each appliance as input. For example, if the air conditioner is running, it records that the power consumption is 1.5 kWh per hour. The collected electricity usage data is obtained as output.

[1834] Step 2: Save data

[1835] The server stores the collected electricity usage data in a database. As input, the data collected in step 1 is provided. For example, the data is recorded in the format "October 1, 2023 10:00 - Air Conditioner - 1.5kWh". As output, the electricity usage data stored in the database is obtained.

[1836] Step 3: Data analysis

[1837] The server uses data analysis software such as Python or R to analyze the stored data. The electricity usage data stored in the database is provided as input. Specifically, the server extracts electricity usage for specific time periods and for specific home appliances and analyzes usage patterns. For example, it can confirm that air conditioner usage is concentrated between 9:00 AM and 5:00 PM. The analysis results are obtained as output.

[1838] Step 4: Generate power saving suggestions

[1839] The server uses a generative AI model (e.g., GPT-4 (registered trademark)) to generate power-saving suggestions based on the results of the data analysis. The analysis results obtained in step 3 are provided as input. The following prompt sentence is input to the generative AI model: "Please generate power-saving suggestions for the user based on the following data. Data: Air conditioner usage time: 9:00 AM to 5:00 PM, usage frequency: 5 times a day. Suggestion: How to optimize air conditioner usage time." The generated power-saving suggestions are obtained as output.

[1840] Step 5: Present your proposal

[1841] The server sends the generated power-saving suggestions to the user's device. As input, the power-saving suggestions generated in step 4 are provided. The user can check the suggestions through their smartphone or computer. For example, a notification may be displayed saying, "You can reduce your electricity usage by reducing your air conditioner usage between 9:00 AM and 5:00 PM." As output, the power-saving suggestions presented to the user are obtained.

[1842] (Application example 2)

[1843] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1844] Conventional electricity usage monitoring systems are often only applied to household appliances, making it difficult to optimize electricity usage and propose energy-saving measures in physical stores. In addition, there was a lack of concrete means for store managers to grasp electricity usage in real time and take appropriate energy-saving measures. As a result, the efficiency of electricity consumption in physical stores was not fully achieved.

[1845] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for linking the generative AI with a smart home appliance control app and a smart home assistant, means for realizing automatic home appliance control, means for realizing electricity usage monitoring, means for inquiring about electricity usage by voice or text, means for monitoring the electricity usage of each home appliance in the store in real time, means for analyzing usage patterns of specific time periods and specific home appliances, means for notifying the store manager of energy-saving suggestions, and means for displaying past energy usage data in graphs to visualize the energy-saving effects. This enables optimization of electricity usage in physical stores and efficient energy-saving measures.

[1846] "Generative AI" is a type of artificial intelligence that generates new information and predictions based on data.

[1847] A "smart home appliance control app" is an application that allows you to remotely control home appliances using devices such as smartphones and tablets.

[1848] A "smart home assistant" is a device or software that uses voice recognition and artificial intelligence to comprehensively manage and control home appliances and systems.

[1849] "Automatic home appliance control" is a function that automatically operates and manages home appliances based on user settings and behavioral patterns.

[1850] "Electricity usage monitoring" is a function that measures and records the electricity consumed by home appliances and systems in real time.

[1851] "A means for querying electricity usage via voice or text" is an interface that allows a user to obtain information about electricity usage via voice commands or text input.

[1852] "Means for monitoring the electricity usage of each home appliance in the store in real time" is a function that measures the electricity consumption of home appliances installed in a physical store in real time and collects data.

[1853] "Means for analyzing usage patterns during specific time periods or for specific home appliances" refers to a function that analyzes usage trends during specific time periods or for specific home appliances based on collected electricity consumption data.

[1854] The "means for notifying the store manager of power saving suggestions" is a function that notifies the store manager of methods for reducing power consumption and optimal usage patterns based on the analysis results.

[1855] "A means of displaying past electricity usage data in a graph and visualizing the effects of energy saving" is a function that visually displays power consumption data collected in the past and clearly shows the effects of energy saving.

[1856] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[1857] System Configuration

[1858] This system uses the following hardware and software:

[1859] Hardware:

[1860] Smart meter: Measures the electricity usage of each appliance in the store in real time.

[1861] Server: Collects, stores, and analyzes data.

[1862] Smartphone or tablet: A device used by the store manager to operate the application.

[1863] software:

[1864] Python: A programming language for data analysis and visualization.

[1865] Pandas: A library for data preprocessing and aggregation.

[1866] Matplotlib: A library for generating graphs.

[1867] Smart home appliance control app: An application for remotely controlling home appliances.

[1868] Smart home assistant: Software that manages and controls home appliances and systems in an integrated manner.

[1869] Data processing and calculation

[1870] The server receives electricity usage data collected from smart meters in real time and stores it in a database. Next, it preprocesses the data using Pandas to analyze usage patterns for specific time periods and specific appliances. The analysis results are visualized as graphs using Matplotlib.

[1871] Generative AI learns user behavior patterns from collected data, automates home appliance control based on predictions, and generates energy-saving suggestions that are notified to store managers via a smart home appliance control app.

[1872] Specific examples

[1873] For example, if air conditioner usage peaks at 2 p.m., the generative AI will generate a power-saving suggestion such as, "Air conditioner usage peaks at 2 p.m. Consider shifting usage to the morning." This suggestion is then notified to the store manager via the smart home appliance control app.

[1874] Example prompt sentence:

[1875] "Peak air conditioning usage time is 2 p.m. Consider shifting usage to the morning."

[1876] In this way, it becomes possible to optimize electricity usage in physical stores and take efficient energy-saving measures.

[1877] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1878] Step 1:

[1879] The server receives real-time electricity usage data from the smart meter. The input is the electricity usage data from the smart meter, and the output is the raw data stored in the database in the server. Specifically, the server receives data packets sent from the smart meter and records them in the database.

[1880] Step 2:

[1881] The server uses Pandas to preprocess the received electricity usage data. The input is the raw data saved in step 1, and the output is the preprocessed clean data. Specifically, the server completes missing values ​​in the data and aligns the data based on timestamps.

[1882] Step 3:

[1883] The server analyzes the usage patterns of specific time periods and specific appliances based on the preprocessed data. The input is the clean data obtained in step 2, and the output is the analysis results of the usage patterns. Specifically, the server uses Pandas to group the data and aggregate the usage of each appliance by time period.

[1884] Step 4:

[1885] The server uses generative AI to learn user behavior patterns and automates home appliance control based on predictions. The input is the analysis results of usage patterns obtained in step 3, and the output is the automated home appliance control settings. Specifically, the server uses a machine learning algorithm to model behavior patterns and generate an optimal home appliance control schedule.

[1886] Step 5:

[1887] The server generates power-saving suggestions using generative AI and notifies the store manager through the smart home appliance control app. The input is the automated home appliance control settings obtained in step 4, and the output is a notification of the power-saving suggestions. Specifically, the server generates the power-saving suggestions using the generative AI model and sends them to the app as prompts.

[1888] Step 6:

[1889] The terminal (smartphone or tablet) receives the power-saving suggestion sent from the server and notifies the store manager. The input is the notification of the power-saving suggestion generated in step 5, and the output is a notification to the store manager. Specifically, the terminal displays the power-saving suggestion using the push notification function.

[1890] Step 7:

[1891] The user (store manager) checks the power-saving suggestions through the terminal and adjusts the appliance usage schedule as necessary. The input is the power-saving suggestions received in step 6, and the output is the adjusted appliance usage schedule. Specifically, the user operates the app interface to change the appliance settings.

[1892] Step 8:

[1893] The server displays past electricity usage data in a graph to visualize the power saving effect. The input is the clean data obtained in step 2 and the analysis results of the usage pattern obtained in step 3, and the output is the power saving effect displayed as a graph. Specifically, the server uses Matplotlib to visualize the data and generate the graph.

[1894] Example 3

[1895] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1896] In conventional smart home systems, users had to manually open a dedicated application to check their electricity usage. Furthermore, when using a voice assistant to inquire about electricity usage, it was difficult to obtain accurate data or generate natural responses. Furthermore, there was a lack of technology to accurately understand the user's intent and provide appropriate responses. This resulted in a poor user experience and impaired the convenience of smart home systems.

[1897] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1898] In this invention, the server includes means for converting voice to text, means for analyzing the text and understanding the user's intent, means for acquiring electricity usage data, means for generating a response based on the acquired data, and means for conveying the generated response to the user by voice. This makes it possible to acquire accurate data and generate a natural response when a user inquires about electricity usage by voice or text.

[1899] "Generative AI" is an artificial intelligence technology that analyzes user input and generates answers and predictions in natural language.

[1900] A "smart home appliance control app" is application software that allows you to remotely control home appliances using devices such as smartphones and tablets.

[1901] A "smart home assistant" is a device that uses voice recognition and natural language processing to respond to users' voice commands and control home appliances and systems.

[1902] "Automatic home appliance control" is a function that automatically operates home appliances based on user settings and behavior patterns.

[1903] "Electricity usage monitoring" is a function that monitors and records the amount of electricity consumed within the home in real time.

[1904] A "speech-to-text means" is a technology for converting a user's voice input into textual information.

[1905] "Means for analyzing text and understanding user intent" refers to technology that analyzes text data to understand what the user is looking for.

[1906] "Means for acquiring electricity usage data" refers to a technique for collecting electricity consumption data from the monitoring system.

[1907] "Means for generating answers based on acquired data" refers to technology for analyzing collected data and generating appropriate answers to user inquiries.

[1908] The "means for conveying the generated answer to the user by voice" is a technology for converting the text-format answer into voice and conveying it to the user.

[1909] MODE FOR CARRYING OUT THE INVENTION

[1910] The present invention relates to a smart home system that allows users to inquire about electricity usage by voice or text. Specific embodiments of this system will be described below.

[1911] System configuration

[1912] The system consists of the following major components:

[1913] 1. Smart Home Assistant (Device)

[1914] 2. Server

[1915] 3. Monitoring System

[1916] Hardware and software used

[1917] Speech Recognition Software: Google® Cloud Speech-to-Text API

[1918] Generative AI model: GPT-4(registered trademark) by OpenAI(registered trademark)

[1919] Text-to-speech software: Amazon Polly or Google® Cloud Text-to-Speech

[1920] Monitoring systems: smart meters and energy monitoring systems

[1921] Program processing

[1922] When a user asks the smart home assistant, "How much electricity have I used this month?", the following process takes place:

[1923] 1. Speech-to-text

[1924] The device records the user's voice and sends the voice data to the Google® Cloud Speech-to-Text API, which converts the voice into text and returns it to the device.

[1925] 2. Sending a text

[1926] The terminal sends the converted text to the server. The text data is sent to the server as an HTTP request.

[1927] 3. Text Analysis

[1928] The server analyzes the received text and uses a generative AI model (OpenAI's GPT-4®) to understand the user's intent. The generative AI model generates an appropriate prompt.

[1929] 4. Obtaining electricity usage data

[1930] The server calls the monitoring system's API to retrieve electricity usage data, and may also retrieve past data from a database.

[1931] 5. Answer Generation

[1932] The server uses the electricity usage data to generate answers to user queries, again in natural language using a generative AI model.

[1933] 6. Sending answers and voice synthesis

[1934] The server sends the generated answer to the device, which then verbally communicates the answer to the user. The voice synthesis uses Amazon Polly...

Claims

1. A means of linking generative artificial intelligence with home appliance control applications and home assistants; A means for realizing automatic home appliance control; A means for implementing monitoring of electricity usage; A means for collecting electricity usage data; A means for storing the collected data; and A means for analyzing the stored data; means for generating energy saving suggestions based on the analysis results; means for presenting the generated power saving suggestions to a user; The generative AI learns user behavior patterns and automates home appliance control based on predictions. system.

2. A means of linking generative AI with smart home appliance control apps and smart home assistants; A means for realizing automatic home appliance control; A means for implementing monitoring of electricity usage; A means to monitor the electricity usage of each appliance in the store in real time, A means for analyzing usage patterns of specific time periods or specific home appliances based on the monitored electricity usage, and A means for notifying a store manager of energy saving suggestions based on the analysis results; A means to display past electricity usage data in a graph and visualize the effects of energy savings, A system including:

3. A means to link generative AI with smart home appliance control apps and smart home assistants, A means for realizing automatic home appliance control; A means for implementing monitoring of electricity usage; A means to inquire about electricity usage by voice or text, A means for converting speech to text; A means to analyze text and understand user intent; A means for acquiring electricity usage data based on the monitored electricity usage; A means for generating responses to user inquiries based on the acquired data; and A means of communicating the generated answer to the user; and A system including:

Images